Issues with viewing and exporting data (Aggregate & Briefcase)

Hello ODK Community,

I am running Aggregate 1.4.11 (upgraded from 1.4.9 a few days ago, without
issues)

Quite similar to the report in the following message:
https://groups.google.com/forum/#!topic/opendatakit/Z6n0bHFdSbM
I get the following error messages:

  1. When accessing my Aggregate server hosted on Google Cloud -> "Error:
    Problem persisting data or accessing data (Somehow DB entities for
    publisher got into problem state)"

  2. When pulling data from the server with Briefcase (only some instances
    are concerned; for only one form out of 2) -> "fetching instance 289 ...
    fetching instance 290 ... SUBMISSION NOT RETRIEVED: Error fetching
    submission uri: uuid:20a0ca9c-4157-4caf-935f-ec65a7bbc4a7 details: Fetch of
    a submission failed. Detailed error: Internal Server Error (500) while
    accessing:
    https://enuff-baseline.appspot.com/view/downloadSubmission?formId=enuff_farmHH[%40version%3Dnull+and+%40uiVersion%3Dnull]%2FQuestionnaire_Farming_households[%40key%3Duuid%3A20a0ca9c-4157-4caf-935f-ec65a7bbc4a7]
    Please verify that the URL, your user credentials and your permissions are
    all correct."

  3. When looking at the AppEngine log that corresponds to the above
    Briefcase activity -> "org.opendatakit.aggregate.submission.Submission
    fetchSubmission: Unable to reconstruct submission for
    opendatakit.NFFARMHH_CORE uri uuid:20a0ca9c-4157-4caf-935f-ec65a7bbc4a7"

I did follow the instructions at
https://github.com/opendatakit/opendatakit/wiki/Aggregate-AppEngine-Troubleshooting,
actually found one corrupted filled-in form submission and repaired it. I
investigated but didn't find any issues with the form definition table.

Any clues?

Thanks in advance.

GL

Hi Guillaume,

If you've followed the steps at
https://github.com/opendatakit/opendatakit/wiki/Aggregate-AppEngine-Troubleshooting#repairing-a-filled-in-form-submission,
you should be fine going forward.

Just to be sure, re-run Briefcase on the command line for that
particular form, then check the logs again to see if there is an
error. If there is, you likely still have some corrupted submissions
for whatever the UUID is.

In your Datastore view on App Engine, you'll probably see a couple of
tables that start with NFFARMHH. Make sure you've checked for
duplicates in each of those tables.

Yaw

··· -- Need ODK consultants? Nafundi provides form design, server setup, in-field training, and software development for ODK. Go to https://nafundi.com to get started.

On Fri, Aug 19, 2016 at 6:15 AM, Guillaume Lestrelin g.lestrelin@gmail.com wrote:

Hello ODK Community,

I am running Aggregate 1.4.11 (upgraded from 1.4.9 a few days ago, without
issues)

Quite similar to the report in the following message:
https://groups.google.com/forum/#!topic/opendatakit/Z6n0bHFdSbM
I get the following error messages:

  1. When accessing my Aggregate server hosted on Google Cloud -> "Error:
    Problem persisting data or accessing data (Somehow DB entities for publisher
    got into problem state)"

  2. When pulling data from the server with Briefcase (only some instances are
    concerned; for only one form out of 2) -> "fetching instance 289 ...
    fetching instance 290 ... SUBMISSION NOT RETRIEVED: Error fetching
    submission uri: uuid:20a0ca9c-4157-4caf-935f-ec65a7bbc4a7 details: Fetch of
    a submission failed. Detailed error: Internal Server Error (500) while
    accessing:
    https://enuff-baseline.appspot.com/view/downloadSubmission?formId=enuff_farmHH[%40version%3Dnull+and+%40uiVersion%3Dnull]%2FQuestionnaire_Farming_households[%40key%3Duuid%3A20a0ca9c-4157-4caf-935f-ec65a7bbc4a7]
    Please verify that the URL, your user credentials and your permissions are
    all correct."

  3. When looking at the AppEngine log that corresponds to the above Briefcase
    activity -> "org.opendatakit.aggregate.submission.Submission
    fetchSubmission: Unable to reconstruct submission for
    opendatakit.NFFARMHH_CORE uri uuid:20a0ca9c-4157-4caf-935f-ec65a7bbc4a7"

I did follow the instructions at
https://github.com/opendatakit/opendatakit/wiki/Aggregate-AppEngine-Troubleshooting,
actually found one corrupted filled-in form submission and repaired it. I
investigated but didn't find any issues with the form definition table.

Any clues?

Thanks in advance.

GL

--

Post: opendatakit@googlegroups.com
Unsubscribe: opendatakit+unsubscribe@googlegroups.com
Options: http://groups.google.com/group/opendatakit?hl=en


You received this message because you are subscribed to the Google Groups
"ODK Community" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to opendatakit+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

I didn't think about going through the different tables (just did the
'Query by GCL' thing) and, indeed, I find a lot of duplicates...
Many thanks for your advice Yaw.

GL

Just a complementary question as I find a lot of duplicates, in a lot of
different tables, for a lot of UUIDs, really a lot...

Rather than spending days retrieving/repairing corrupted submissions in the
AppEngine, would it be possible and could it be more efficient to wait for
the enumerators to come back from the field and use Briefcase to pull the
completed forms directly from the tablets + reconstruct the whole database
afterwards (pushing forms into an empty Aggregate database)?

Thanks in advance for any advice/feedback.

GL

Guillaume,

In my experience, even on massive campaigns, there have only been a
couple of duplicate rows. Make sure that you are checking for
uniqueness by looking at the _ORDINAL_NUMBER column.

And yes, if you are sure all your enumerators have all the data, you
can use Briefcase to pull the data and push it wherever. Just make
sure the forms are completed because Briefcase pulls incomplete forms
as well.

Yaw

··· -- Need ODK consultants? Nafundi provides form design, server setup, in-field training, and software development for ODK. Go to https://nafundi.com to get started.

On Fri, Aug 19, 2016 at 9:13 AM, Guillaume Lestrelin g.lestrelin@gmail.com wrote:

Just a complementary question as I find a lot of duplicates, in a lot of
different tables, for a lot of UUIDs, really a lot...

Rather than spending days retrieving/repairing corrupted submissions in the
AppEngine, would it be possible and could it be more efficient to wait for
the enumerators to come back from the field and use Briefcase to pull the
completed forms directly from the tablets + reconstruct the whole database
afterwards (pushing forms into an empty Aggregate database)?

Thanks in advance for any advice/feedback.

GL

--

Post: opendatakit@googlegroups.com
Unsubscribe: opendatakit+unsubscribe@googlegroups.com
Options: http://groups.google.com/group/opendatakit?hl=en


You received this message because you are subscribed to the Google Groups
"ODK Community" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to opendatakit+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Thanks Yaw,

Actually, Briefcase returns errors "fetching submission" for a total of 51
UUIDs (out of 1200+ so far). Looking at the Datastore, I find duplicate
rows for all these UUIDs in something like 10-12 tables. I wonder about the
underlying cause: AppEngine daily/free quotas issues? fairly slow internet
connection in the field? Too many repeat groups in the form?
I also wonder: what about deleting the corresponding 51 submissions (entire
completed forms/tables) directly in Aggregate or in the Datatore? What
would be the behaviour of ODK Collect? Would the corresponding forms be re
sent or not?

Cheers,

Yom

Guillaume,

"This is occasionally necessary because Google can kill the server
process...These failures are more likely to occur if you are trying to
run using just free quota, or if you are heavily using your system and
the form being uploaded is large or contains many repeat groups."

I would guess that deleting the submissions in Aggregate would solve
the problem of errors. I would also guess that you could resend those
forms successfully because the UUID that Aggregate uses to check
duplicates won't be on the server. I say guess because I didn't write
that code. Please test to confirm.

Going forward the best way to prevent these errors is to run 1.4.11
and enable billing.

Yaw

··· -- Need ODK consultants? Nafundi provides form design, server setup, in-field training, and software development for ODK. Go to https://nafundi.com to get started.

On Mon, Aug 22, 2016 at 7:57 AM, Guillaume Lestrelin g.lestrelin@gmail.com wrote:

Thanks Yaw,

Actually, Briefcase returns errors "fetching submission" for a total of 51
UUIDs (out of 1200+ so far). Looking at the Datastore, I find duplicate rows
for all these UUIDs in something like 10-12 tables. I wonder about the
underlying cause: AppEngine daily/free quotas issues? fairly slow internet
connection in the field? Too many repeat groups in the form?
I also wonder: what about deleting the corresponding 51 submissions (entire
completed forms/tables) directly in Aggregate or in the Datatore? What would
be the behaviour of ODK Collect? Would the corresponding forms be re sent or
not?

Cheers,

Yom

--

Post: opendatakit@googlegroups.com
Unsubscribe: opendatakit+unsubscribe@googlegroups.com
Options: http://groups.google.com/group/opendatakit?hl=en


You received this message because you are subscribed to the Google Groups
"ODK Community" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to opendatakit+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Yaw,

Deleting the submissions in Aggregate is not an option apparently since my
attempts to display / filter the problematic submissions (using their
UUIDs) result in error messages like: "Error: Problem persisting data or
accessing data (SELECT * FROM
NFFARMHH_GROUP_OTHER_MEMBERS_GROUP_OTHER_GENDER_AGE WHERE _TOP_LEVEL_AURI =
uuid:46d2067c-4315-4bfc-9ec6-dd4c04027279 AND _PARENT_AURI =
uuid:46d2067c-4315-4bfc-9ec6-dd4c04027279 is missing a (repeat) group
instance OR has an extra copies.)"

In order to test your suggestion, I deleted a few problematic submissions
directly in the Datastore but to do so, I have to run queries and delete
both parent and child entities (13 different tables) manually. From what I
understand (I am not exactly a computer expert :-P), the "low-level Java
Datastore API" does not deal with child entities when deleting parent
entities... If I leave these child entities, I suppose this could create
new issues when resending the forms to Aggregate, right? Any suggestions of
a faster/cleaner/more secure method to delete the whole 'parent-child'
groups?

Yom

··· Le lundi 22 août 2016 20:49:32 UTC+7, Yaw Anokwa a écrit : > > Guillaume, > > "This is occasionally necessary because Google can kill the server > process...These failures are more likely to occur if you are trying to > run using just free quota, or if you are heavily using your system and > the form being uploaded is large or contains many repeat groups." > > https://github.com/opendatakit/opendatakit/wiki/Aggregate-AppEngine-Troubleshooting#repairing-your-app-engine-database-yourself > > I would guess that deleting the submissions in Aggregate would solve > the problem of errors. I would also guess that you could resend those > forms successfully because the UUID that Aggregate uses to check > duplicates won't be on the server. I say guess because I didn't write > that code. Please test to confirm. > > Going forward the best way to prevent these errors is to run 1.4.11 > and enable billing. > > Yaw > -- > Need ODK consultants? Nafundi provides form design, server setup, > in-field training, and software development for ODK. Go to > https://nafundi.com to get started. > > On Mon, Aug 22, 2016 at 7:57 AM, Guillaume Lestrelin <g.les...@gmail.com > wrote: > > Thanks Yaw, > > > > Actually, Briefcase returns errors "fetching submission" for a total of > 51 > > UUIDs (out of 1200+ so far). Looking at the Datastore, I find duplicate > rows > > for all these UUIDs in something like 10-12 tables. I wonder about the > > underlying cause: AppEngine daily/free quotas issues? fairly slow > internet > > connection in the field? Too many repeat groups in the form? > > I also wonder: what about deleting the corresponding 51 submissions > (entire > > completed forms/tables) directly in Aggregate or in the Datatore? What > would > > be the behaviour of ODK Collect? Would the corresponding forms be re > sent or > > not? > > > > Cheers, > > > > Yom > > > > -- > > -- > > Post: opend...@googlegroups.com > > Unsubscribe: opendatakit...@googlegroups.com > > Options: http://groups.google.com/group/opendatakit?hl=en > > > > --- > > You received this message because you are subscribed to the Google > Groups > > "ODK Community" group. > > To unsubscribe from this group and stop receiving emails from it, send > an > > email to opendatakit...@googlegroups.com . > > For more options, visit https://groups.google.com/d/optout. >

Yom,

This is an issue that we'll have to escalate to Mitch and Waylon to
see if they have any ideas. I know they are both busy, but they will
respond when they get a moment...

Thanks,

Yaw

··· -- Need ODK consultants? Nafundi provides form design, server setup, in-field training, and software development for ODK. Go to https://nafundi.com to get started.

On Wed, Aug 24, 2016 at 9:18 AM, Guillaume Lestrelin g.lestrelin@gmail.com wrote:

Yaw,

Deleting the submissions in Aggregate is not an option apparently since my
attempts to display / filter the problematic submissions (using their UUIDs)
result in error messages like: "Error: Problem persisting data or accessing
data (SELECT * FROM NFFARMHH_GROUP_OTHER_MEMBERS_GROUP_OTHER_GENDER_AGE
WHERE _TOP_LEVEL_AURI = uuid:46d2067c-4315-4bfc-9ec6-dd4c04027279 AND
_PARENT_AURI = uuid:46d2067c-4315-4bfc-9ec6-dd4c04027279 is missing a
(repeat) group instance OR has an extra copies.)"

In order to test your suggestion, I deleted a few problematic submissions
directly in the Datastore but to do so, I have to run queries and delete
both parent and child entities (13 different tables) manually. From what I
understand (I am not exactly a computer expert :-P), the "low-level Java
Datastore API" does not deal with child entities when deleting parent
entities... If I leave these child entities, I suppose this could create new
issues when resending the forms to Aggregate, right? Any suggestions of a
faster/cleaner/more secure method to delete the whole 'parent-child' groups?

Yom

Le lundi 22 août 2016 20:49:32 UTC+7, Yaw Anokwa a écrit :

Guillaume,

"This is occasionally necessary because Google can kill the server
process...These failures are more likely to occur if you are trying to
run using just free quota, or if you are heavily using your system and
the form being uploaded is large or contains many repeat groups."

https://github.com/opendatakit/opendatakit/wiki/Aggregate-AppEngine-Troubleshooting#repairing-your-app-engine-database-yourself

I would guess that deleting the submissions in Aggregate would solve
the problem of errors. I would also guess that you could resend those
forms successfully because the UUID that Aggregate uses to check
duplicates won't be on the server. I say guess because I didn't write
that code. Please test to confirm.

Going forward the best way to prevent these errors is to run 1.4.11
and enable billing.

Yaw

Need ODK consultants? Nafundi provides form design, server setup,
in-field training, and software development for ODK. Go to
https://nafundi.com to get started.

On Mon, Aug 22, 2016 at 7:57 AM, Guillaume Lestrelin g.les...@gmail.com wrote:

Thanks Yaw,

Actually, Briefcase returns errors "fetching submission" for a total of
51
UUIDs (out of 1200+ so far). Looking at the Datastore, I find duplicate
rows
for all these UUIDs in something like 10-12 tables. I wonder about the
underlying cause: AppEngine daily/free quotas issues? fairly slow
internet
connection in the field? Too many repeat groups in the form?
I also wonder: what about deleting the corresponding 51 submissions
(entire
completed forms/tables) directly in Aggregate or in the Datatore? What
would
be the behaviour of ODK Collect? Would the corresponding forms be re
sent or
not?

Cheers,

Yom

--

Post: opend...@googlegroups.com
Unsubscribe: opendatakit...@googlegroups.com
Options: http://groups.google.com/group/opendatakit?hl=en


You received this message because you are subscribed to the Google
Groups
"ODK Community" group.
To unsubscribe from this group and stop receiving emails from it, send
an
email to opendatakit...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--

Post: opendatakit@googlegroups.com
Unsubscribe: opendatakit+unsubscribe@googlegroups.com
Options: http://groups.google.com/group/opendatakit?hl=en


You received this message because you are subscribed to the Google Groups
"ODK Community" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to opendatakit+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Automating the data-corruption clean-up is problematic.

If it were to be implemented, it should not happen silently during a
publish or ODK Briefcase pull action (actions that should be guaranteed to
be read-only and non-destructive). Those should still fail and report the
instance ID with the data-corruption issue.

The goal should be to make altering of the data on the server an explicit
user action. This could be via a new button on the Form Management /
Submission Admin sub-tab (e.g., "Repair Submission") that prompted for the
instanceID of the problematic row.

But, implementing that action is tricky because of the many different ways
that failures can occur, with duplicates or omissions of records in any of
the form's tables leading to data corruption. While it is somewhat
straightforward in forms without repeat groups, this gets particularly
complicated when there are repeat groups because those can be arbitrarily
deeply nested.

Given how difficult it has been to generate unique column names in the face
of large deeply-nested forms (there are still forms for which this fails),
I, personally, would not trust any code written to automatically repair a
submission; developing tests to verify the correct handling of all data
corruption failure cases would be similarly impossible.

Perhaps a first step would be for the "Repair Submission" action to
generate a report containing the GQL or SQL statements that users could
cut-and-paste to investigate the errors, confirm the recommendations, and
effect the deletions themselves.

Data corruption should be reduced in likelihood by the changes in ODK
Aggregate 1.4.11 and higher.

··· On Mon, Aug 29, 2016 at 4:00 AM, Yaw Anokwa wrote:

Yom,

This is an issue that we'll have to escalate to Mitch and Waylon to
see if they have any ideas. I know they are both busy, but they will
respond when they get a moment...

Thanks,

Yaw

Need ODK consultants? Nafundi provides form design, server setup,
in-field training, and software development for ODK. Go to
https://nafundi.com to get started.

On Wed, Aug 24, 2016 at 9:18 AM, Guillaume Lestrelin g.lestrelin@gmail.com wrote:

Yaw,

Deleting the submissions in Aggregate is not an option apparently since
my
attempts to display / filter the problematic submissions (using their
UUIDs)
result in error messages like: "Error: Problem persisting data or
accessing
data (SELECT * FROM NFFARMHH_GROUP_OTHER_MEMBERS_GROUP_OTHER_GENDER_AGE
WHERE _TOP_LEVEL_AURI = uuid:46d2067c-4315-4bfc-9ec6-dd4c04027279 AND
_PARENT_AURI = uuid:46d2067c-4315-4bfc-9ec6-dd4c04027279 is missing a
(repeat) group instance OR has an extra copies.)"

In order to test your suggestion, I deleted a few problematic submissions
directly in the Datastore but to do so, I have to run queries and delete
both parent and child entities (13 different tables) manually. From what
I
understand (I am not exactly a computer expert :-P), the "low-level Java
Datastore API" does not deal with child entities when deleting parent
entities... If I leave these child entities, I suppose this could create
new
issues when resending the forms to Aggregate, right? Any suggestions of a
faster/cleaner/more secure method to delete the whole 'parent-child'
groups?

Yom

Le lundi 22 août 2016 20:49:32 UTC+7, Yaw Anokwa a écrit :

Guillaume,

"This is occasionally necessary because Google can kill the server
process...These failures are more likely to occur if you are trying to
run using just free quota, or if you are heavily using your system and
the form being uploaded is large or contains many repeat groups."

https://github.com/opendatakit/opendatakit/wiki/Aggregate-AppEngine-
Troubleshooting#repairing-your-app-engine-database-yourself

I would guess that deleting the submissions in Aggregate would solve
the problem of errors. I would also guess that you could resend those
forms successfully because the UUID that Aggregate uses to check
duplicates won't be on the server. I say guess because I didn't write
that code. Please test to confirm.

Going forward the best way to prevent these errors is to run 1.4.11
and enable billing.

Yaw

Need ODK consultants? Nafundi provides form design, server setup,
in-field training, and software development for ODK. Go to
https://nafundi.com to get started.

On Mon, Aug 22, 2016 at 7:57 AM, Guillaume Lestrelin g.les...@gmail.com wrote:

Thanks Yaw,

Actually, Briefcase returns errors "fetching submission" for a total
of
51
UUIDs (out of 1200+ so far). Looking at the Datastore, I find
duplicate
rows
for all these UUIDs in something like 10-12 tables. I wonder about the
underlying cause: AppEngine daily/free quotas issues? fairly slow
internet
connection in the field? Too many repeat groups in the form?
I also wonder: what about deleting the corresponding 51 submissions
(entire
completed forms/tables) directly in Aggregate or in the Datatore? What
would
be the behaviour of ODK Collect? Would the corresponding forms be re
sent or
not?

Cheers,

Yom

--

Post: opend...@googlegroups.com
Unsubscribe: opendatakit...@googlegroups.com
Options: http://groups.google.com/group/opendatakit?hl=en


You received this message because you are subscribed to the Google
Groups
"ODK Community" group.
To unsubscribe from this group and stop receiving emails from it, send
an
email to opendatakit...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--

Post: opendatakit@googlegroups.com
Unsubscribe: opendatakit+unsubscribe@googlegroups.com
Options: http://groups.google.com/group/opendatakit?hl=en


You received this message because you are subscribed to the Google Groups
"ODK Community" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to opendatakit+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--
Mitch Sundt
Software Engineer
http://www.OpenDataKit.org http://www.opendatakit.org/
University of Washington
mitchellsundt@gmail.com

Thanks for the detailed explanation Mitch.

A more comprehensive "Repair Submission" reporting system would certainly
be useful as Aggregate currently only returns single SQL statements, i.e.
providing info on data corruption issues in only one table whereas
corrupted data might also occur in other tables for the same submission.
This requires the user to either (1) go back and forth between filtering a
corrupted submission in Aggregate and repairing the corrupted tables in the
Datastore or (2) for each corrupted submission, going through every single
table in the Datastore and repairing the corrupted data. I have combined
both options to resolve my issue.

Finally, upgrading is definitely useful. In my case, data corruption seems
to have occurred exclusively before I upgraded to 1.4.11 (I was initially
using 1.4.9).

Cheers,

Yom

··· Le jeudi 8 septembre 2016 01:42:25 UTC+7, Mitchell Sundt a écrit : > > Automating the data-corruption clean-up is problematic. > > If it were to be implemented, it should not happen silently during a > publish or ODK Briefcase pull action (actions that should be guaranteed to > be read-only and non-destructive). Those should still fail and report the > instance ID with the data-corruption issue. > > The goal should be to make altering of the data on the server an explicit > user action. This could be via a new button on the Form Management / > Submission Admin sub-tab (e.g., "Repair Submission") that prompted for the > instanceID of the problematic row. > > But, implementing that action is tricky because of the many different ways > that failures can occur, with duplicates or omissions of records in any of > the form's tables leading to data corruption. While it is somewhat > straightforward in forms without repeat groups, this gets particularly > complicated when there are repeat groups because those can be arbitrarily > deeply nested. > > Given how difficult it has been to generate unique column names in the > face of large deeply-nested forms (there are still forms for which this > fails), I, personally, would not trust any code written to automatically > repair a submission; developing tests to verify the correct handling of all > data corruption failure cases would be similarly impossible. > > Perhaps a first step would be for the "Repair Submission" action to > generate a report containing the GQL or SQL statements that users could > cut-and-paste to investigate the errors, confirm the recommendations, and > effect the deletions themselves. > > Data corruption should be reduced in likelihood by the changes in ODK > Aggregate 1.4.11 and higher. > > > On Mon, Aug 29, 2016 at 4:00 AM, Yaw Anokwa <yan...@nafundi.com > wrote: > >> Yom, >> >> This is an issue that we'll have to escalate to Mitch and Waylon to >> see if they have any ideas. I know they are both busy, but they will >> respond when they get a moment... >> >> Thanks, >> >> Yaw >> -- >> Need ODK consultants? Nafundi provides form design, server setup, >> in-field training, and software development for ODK. Go to >> https://nafundi.com to get started. >> >> On Wed, Aug 24, 2016 at 9:18 AM, Guillaume Lestrelin <g.les...@gmail.com > wrote: >> > Yaw, >> > >> > Deleting the submissions in Aggregate is not an option apparently since >> my >> > attempts to display / filter the problematic submissions (using their >> UUIDs) >> > result in error messages like: "Error: Problem persisting data or >> accessing >> > data (SELECT * FROM NFFARMHH_GROUP_OTHER_MEMBERS_GROUP_OTHER_GENDER_AGE >> > WHERE _TOP_LEVEL_AURI = uuid:46d2067c-4315-4bfc-9ec6-dd4c04027279 AND >> > _PARENT_AURI = uuid:46d2067c-4315-4bfc-9ec6-dd4c04027279 is missing a >> > (repeat) group instance OR has an extra copies.)" >> > >> > In order to test your suggestion, I deleted a few problematic >> submissions >> > directly in the Datastore but to do so, I have to run queries and delete >> > both parent and child entities (13 different tables) manually. From >> what I >> > understand (I am not exactly a computer expert :-P), the "low-level Java >> > Datastore API" does not deal with child entities when deleting parent >> > entities... If I leave these child entities, I suppose this could >> create new >> > issues when resending the forms to Aggregate, right? Any suggestions of >> a >> > faster/cleaner/more secure method to delete the whole 'parent-child' >> groups? >> > >> > Yom >> > >> > >> > Le lundi 22 août 2016 20:49:32 UTC+7, Yaw Anokwa a écrit : >> >> >> >> Guillaume, >> >> >> >> "This is occasionally necessary because Google can kill the server >> >> process...These failures are more likely to occur if you are trying to >> >> run using just free quota, or if you are heavily using your system and >> >> the form being uploaded is large or contains many repeat groups." >> >> >> >> >> https://github.com/opendatakit/opendatakit/wiki/Aggregate-AppEngine-Troubleshooting#repairing-your-app-engine-database-yourself >> >> >> >> I would guess that deleting the submissions in Aggregate would solve >> >> the problem of errors. I would also guess that you could resend those >> >> forms successfully because the UUID that Aggregate uses to check >> >> duplicates won't be on the server. I say guess because I didn't write >> >> that code. Please test to confirm. >> >> >> >> Going forward the best way to prevent these errors is to run 1.4.11 >> >> and enable billing. >> >> >> >> Yaw >> >> -- >> >> Need ODK consultants? Nafundi provides form design, server setup, >> >> in-field training, and software development for ODK. Go to >> >> https://nafundi.com to get started. >> >> >> >> On Mon, Aug 22, 2016 at 7:57 AM, Guillaume Lestrelin wrote: >> >> > Thanks Yaw, >> >> > >> >> > Actually, Briefcase returns errors "fetching submission" for a total >> of >> >> > 51 >> >> > UUIDs (out of 1200+ so far). Looking at the Datastore, I find >> duplicate >> >> > rows >> >> > for all these UUIDs in something like 10-12 tables. I wonder about >> the >> >> > underlying cause: AppEngine daily/free quotas issues? fairly slow >> >> > internet >> >> > connection in the field? Too many repeat groups in the form? >> >> > I also wonder: what about deleting the corresponding 51 submissions >> >> > (entire >> >> > completed forms/tables) directly in Aggregate or in the Datatore? >> What >> >> > would >> >> > be the behaviour of ODK Collect? Would the corresponding forms be re >> >> > sent or >> >> > not? >> >> > >> >> > Cheers, >> >> > >> >> > Yom >> >> > >> >> > -- >> >> > -- >> >> > Post: opend...@googlegroups.com >> >> > Unsubscribe: opendatakit...@googlegroups.com >> >> > Options: http://groups.google.com/group/opendatakit?hl=en >> >> > >> >> > --- >> >> > You received this message because you are subscribed to the Google >> >> > Groups >> >> > "ODK Community" group. >> >> > To unsubscribe from this group and stop receiving emails from it, >> send >> >> > an >> >> > email to opendatakit...@googlegroups.com. >> >> > For more options, visit https://groups.google.com/d/optout. >> > >> > -- >> > -- >> > Post: opend...@googlegroups.com >> > Unsubscribe: opendatakit...@googlegroups.com >> > Options: http://groups.google.com/group/opendatakit?hl=en >> > >> > --- >> > You received this message because you are subscribed to the Google >> Groups >> > "ODK Community" group. >> > To unsubscribe from this group and stop receiving emails from it, send >> an >> > email to opendatakit...@googlegroups.com . >> > For more options, visit https://groups.google.com/d/optout. >> > > > > -- > Mitch Sundt > Software Engineer > http://www.OpenDataKit.org > University of Washington > mitche...@gmail.com > >

Hi Mitch,

Do you have a sense for how much work (hours, days, weeks?) it'd be to
generate such a report?

Thanks,

Yaw

··· -- Need ODK consultants? Nafundi provides form design, server setup, in-field training, and software development for ODK. Go to https://nafundi.com to get started.

On Thu, Sep 8, 2016 at 4:55 AM, Guillaume Lestrelin g.lestrelin@gmail.com wrote:

Thanks for the detailed explanation Mitch.

A more comprehensive "Repair Submission" reporting system would certainly be
useful as Aggregate currently only returns single SQL statements, i.e.
providing info on data corruption issues in only one table whereas corrupted
data might also occur in other tables for the same submission. This requires
the user to either (1) go back and forth between filtering a corrupted
submission in Aggregate and repairing the corrupted tables in the Datastore
or (2) for each corrupted submission, going through every single table in
the Datastore and repairing the corrupted data. I have combined both options
to resolve my issue.

Finally, upgrading is definitely useful. In my case, data corruption seems
to have occurred exclusively before I upgraded to 1.4.11 (I was initially
using 1.4.9).

Cheers,

Yom

Le jeudi 8 septembre 2016 01:42:25 UTC+7, Mitchell Sundt a écrit :

Automating the data-corruption clean-up is problematic.

If it were to be implemented, it should not happen silently during a
publish or ODK Briefcase pull action (actions that should be guaranteed to
be read-only and non-destructive). Those should still fail and report the
instance ID with the data-corruption issue.

The goal should be to make altering of the data on the server an explicit
user action. This could be via a new button on the Form Management /
Submission Admin sub-tab (e.g., "Repair Submission") that prompted for the
instanceID of the problematic row.

But, implementing that action is tricky because of the many different ways
that failures can occur, with duplicates or omissions of records in any of
the form's tables leading to data corruption. While it is somewhat
straightforward in forms without repeat groups, this gets particularly
complicated when there are repeat groups because those can be arbitrarily
deeply nested.

Given how difficult it has been to generate unique column names in the
face of large deeply-nested forms (there are still forms for which this
fails), I, personally, would not trust any code written to automatically
repair a submission; developing tests to verify the correct handling of all
data corruption failure cases would be similarly impossible.

Perhaps a first step would be for the "Repair Submission" action to
generate a report containing the GQL or SQL statements that users could
cut-and-paste to investigate the errors, confirm the recommendations, and
effect the deletions themselves.

Data corruption should be reduced in likelihood by the changes in ODK
Aggregate 1.4.11 and higher.

On Mon, Aug 29, 2016 at 4:00 AM, Yaw Anokwa yan...@nafundi.com wrote:

Yom,

This is an issue that we'll have to escalate to Mitch and Waylon to
see if they have any ideas. I know they are both busy, but they will
respond when they get a moment...

Thanks,

Yaw

Need ODK consultants? Nafundi provides form design, server setup,
in-field training, and software development for ODK. Go to
https://nafundi.com to get started.

On Wed, Aug 24, 2016 at 9:18 AM, Guillaume Lestrelin g.les...@gmail.com wrote:

Yaw,

Deleting the submissions in Aggregate is not an option apparently since
my
attempts to display / filter the problematic submissions (using their
UUIDs)
result in error messages like: "Error: Problem persisting data or
accessing
data (SELECT * FROM NFFARMHH_GROUP_OTHER_MEMBERS_GROUP_OTHER_GENDER_AGE
WHERE _TOP_LEVEL_AURI = uuid:46d2067c-4315-4bfc-9ec6-dd4c04027279 AND
_PARENT_AURI = uuid:46d2067c-4315-4bfc-9ec6-dd4c04027279 is missing a
(repeat) group instance OR has an extra copies.)"

In order to test your suggestion, I deleted a few problematic
submissions
directly in the Datastore but to do so, I have to run queries and
delete
both parent and child entities (13 different tables) manually. From
what I
understand (I am not exactly a computer expert :-P), the "low-level
Java
Datastore API" does not deal with child entities when deleting parent
entities... If I leave these child entities, I suppose this could
create new
issues when resending the forms to Aggregate, right? Any suggestions of
a
faster/cleaner/more secure method to delete the whole 'parent-child'
groups?

Yom

Le lundi 22 août 2016 20:49:32 UTC+7, Yaw Anokwa a écrit :

Guillaume,

"This is occasionally necessary because Google can kill the server
process...These failures are more likely to occur if you are trying to
run using just free quota, or if you are heavily using your system and
the form being uploaded is large or contains many repeat groups."

https://github.com/opendatakit/opendatakit/wiki/Aggregate-AppEngine-Troubleshooting#repairing-your-app-engine-database-yourself

I would guess that deleting the submissions in Aggregate would solve
the problem of errors. I would also guess that you could resend those
forms successfully because the UUID that Aggregate uses to check
duplicates won't be on the server. I say guess because I didn't write
that code. Please test to confirm.

Going forward the best way to prevent these errors is to run 1.4.11
and enable billing.

Yaw

Need ODK consultants? Nafundi provides form design, server setup,
in-field training, and software development for ODK. Go to
https://nafundi.com to get started.

On Mon, Aug 22, 2016 at 7:57 AM, Guillaume Lestrelin g.les...@gmail.com wrote:

Thanks Yaw,

Actually, Briefcase returns errors "fetching submission" for a total
of
51
UUIDs (out of 1200+ so far). Looking at the Datastore, I find
duplicate
rows
for all these UUIDs in something like 10-12 tables. I wonder about
the
underlying cause: AppEngine daily/free quotas issues? fairly slow
internet
connection in the field? Too many repeat groups in the form?
I also wonder: what about deleting the corresponding 51 submissions
(entire
completed forms/tables) directly in Aggregate or in the Datatore?
What
would
be the behaviour of ODK Collect? Would the corresponding forms be re
sent or
not?

Cheers,

Yom

--

Post: opend...@googlegroups.com
Unsubscribe: opendatakit...@googlegroups.com
Options: http://groups.google.com/group/opendatakit?hl=en


You received this message because you are subscribed to the Google
Groups
"ODK Community" group.
To unsubscribe from this group and stop receiving emails from it,
send
an
email to opendatakit...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--

Post: opend...@googlegroups.com
Unsubscribe: opendatakit...@googlegroups.com
Options: http://groups.google.com/group/opendatakit?hl=en


You received this message because you are subscribed to the Google
Groups
"ODK Community" group.
To unsubscribe from this group and stop receiving emails from it, send
an
email to opendatakit...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--
Mitch Sundt
Software Engineer
http://www.OpenDataKit.org
University of Washington
mitche...@gmail.com

Various parts to this:

4 hr: The plumbing between the GWT-based UI (new button, new pop-up dialog)
and a servlet for this, and the changes to the web.xml and security
configuration to show everything is about 4 hours (an example would be the
upload-permissions-csv feature in 1.4.12; note: that is spread across
several check-ins).

3 days: I would push the GQL / SQL syntax construction down into the
persistence engine implementations. This is already what is done internally
with queries in the PostgreSQL / MySQL case, so you'd just need to grab the
query string without executing the query. For Google, you would need to add
that functionality (since it is all API-based). Probably about 2-3 days
for this to understand the layers, add the new APIs and write the Google
GQL syntax writer.

3-10 days: Writing the servlet that would lay out HTML content with details
on a specific instance ID within a table. It depends upon how fancy you
want to make the page and making sure you handle all the shadow,
select-multiple, repeat group, and attachment tables correctly.

··· On Thu, Sep 8, 2016 at 2:42 AM, Yaw Anokwa wrote:

Hi Mitch,

Do you have a sense for how much work (hours, days, weeks?) it'd be to
generate such a report?

Thanks,

Yaw

Need ODK consultants? Nafundi provides form design, server setup,
in-field training, and software development for ODK. Go to
https://nafundi.com to get started.

On Thu, Sep 8, 2016 at 4:55 AM, Guillaume Lestrelin g.lestrelin@gmail.com wrote:

Thanks for the detailed explanation Mitch.

A more comprehensive "Repair Submission" reporting system would
certainly be
useful as Aggregate currently only returns single SQL statements, i.e.
providing info on data corruption issues in only one table whereas
corrupted
data might also occur in other tables for the same submission. This
requires
the user to either (1) go back and forth between filtering a corrupted
submission in Aggregate and repairing the corrupted tables in the
Datastore
or (2) for each corrupted submission, going through every single table in
the Datastore and repairing the corrupted data. I have combined both
options
to resolve my issue.

Finally, upgrading is definitely useful. In my case, data corruption
seems
to have occurred exclusively before I upgraded to 1.4.11 (I was initially
using 1.4.9).

Cheers,

Yom

Le jeudi 8 septembre 2016 01:42:25 UTC+7, Mitchell Sundt a écrit :

Automating the data-corruption clean-up is problematic.

If it were to be implemented, it should not happen silently during a
publish or ODK Briefcase pull action (actions that should be guaranteed
to
be read-only and non-destructive). Those should still fail and report
the
instance ID with the data-corruption issue.

The goal should be to make altering of the data on the server an
explicit
user action. This could be via a new button on the Form Management /
Submission Admin sub-tab (e.g., "Repair Submission") that prompted for
the
instanceID of the problematic row.

But, implementing that action is tricky because of the many different
ways
that failures can occur, with duplicates or omissions of records in any
of
the form's tables leading to data corruption. While it is somewhat
straightforward in forms without repeat groups, this gets particularly
complicated when there are repeat groups because those can be
arbitrarily
deeply nested.

Given how difficult it has been to generate unique column names in the
face of large deeply-nested forms (there are still forms for which this
fails), I, personally, would not trust any code written to automatically
repair a submission; developing tests to verify the correct handling of
all
data corruption failure cases would be similarly impossible.

Perhaps a first step would be for the "Repair Submission" action to
generate a report containing the GQL or SQL statements that users could
cut-and-paste to investigate the errors, confirm the recommendations,
and
effect the deletions themselves.

Data corruption should be reduced in likelihood by the changes in ODK
Aggregate 1.4.11 and higher.

On Mon, Aug 29, 2016 at 4:00 AM, Yaw Anokwa yan...@nafundi.com wrote:

Yom,

This is an issue that we'll have to escalate to Mitch and Waylon to
see if they have any ideas. I know they are both busy, but they will
respond when they get a moment...

Thanks,

Yaw

Need ODK consultants? Nafundi provides form design, server setup,
in-field training, and software development for ODK. Go to
https://nafundi.com to get started.

On Wed, Aug 24, 2016 at 9:18 AM, Guillaume Lestrelin g.les...@gmail.com wrote:

Yaw,

Deleting the submissions in Aggregate is not an option apparently
since
my
attempts to display / filter the problematic submissions (using their
UUIDs)
result in error messages like: "Error: Problem persisting data or
accessing
data (SELECT * FROM NFFARMHH_GROUP_OTHER_MEMBERS_
GROUP_OTHER_GENDER_AGE
WHERE _TOP_LEVEL_AURI = uuid:46d2067c-4315-4bfc-9ec6-dd4c04027279
AND
_PARENT_AURI = uuid:46d2067c-4315-4bfc-9ec6-dd4c04027279 is missing
a
(repeat) group instance OR has an extra copies.)"

In order to test your suggestion, I deleted a few problematic
submissions
directly in the Datastore but to do so, I have to run queries and
delete
both parent and child entities (13 different tables) manually. From
what I
understand (I am not exactly a computer expert :-P), the "low-level
Java
Datastore API" does not deal with child entities when deleting parent
entities... If I leave these child entities, I suppose this could
create new
issues when resending the forms to Aggregate, right? Any suggestions
of
a
faster/cleaner/more secure method to delete the whole 'parent-child'
groups?

Yom

Le lundi 22 août 2016 20:49:32 UTC+7, Yaw Anokwa a écrit :

Guillaume,

"This is occasionally necessary because Google can kill the server
process...These failures are more likely to occur if you are trying
to
run using just free quota, or if you are heavily using your system
and
the form being uploaded is large or contains many repeat groups."

https://github.com/opendatakit/opendatakit/wiki/
Aggregate-AppEngine-Troubleshooting#repairing-your-app-engine-database-
yourself

I would guess that deleting the submissions in Aggregate would solve
the problem of errors. I would also guess that you could resend
those
forms successfully because the UUID that Aggregate uses to check
duplicates won't be on the server. I say guess because I didn't
write
that code. Please test to confirm.

Going forward the best way to prevent these errors is to run 1.4.11
and enable billing.

Yaw

Need ODK consultants? Nafundi provides form design, server setup,
in-field training, and software development for ODK. Go to
https://nafundi.com to get started.

On Mon, Aug 22, 2016 at 7:57 AM, Guillaume Lestrelin g.les...@gmail.com wrote:

Thanks Yaw,

Actually, Briefcase returns errors "fetching submission" for a
total
of
51
UUIDs (out of 1200+ so far). Looking at the Datastore, I find
duplicate
rows
for all these UUIDs in something like 10-12 tables. I wonder about
the
underlying cause: AppEngine daily/free quotas issues? fairly slow
internet
connection in the field? Too many repeat groups in the form?
I also wonder: what about deleting the corresponding 51
submissions
(entire
completed forms/tables) directly in Aggregate or in the Datatore?
What
would
be the behaviour of ODK Collect? Would the corresponding forms be
re
sent or
not?

Cheers,

Yom

--

Post: opend...@googlegroups.com
Unsubscribe: opendatakit...@googlegroups.com
Options: http://groups.google.com/group/opendatakit?hl=en


You received this message because you are subscribed to the Google
Groups
"ODK Community" group.
To unsubscribe from this group and stop receiving emails from it,
send
an
email to opendatakit...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--

Post: opend...@googlegroups.com
Unsubscribe: opendatakit...@googlegroups.com
Options: http://groups.google.com/group/opendatakit?hl=en


You received this message because you are subscribed to the Google
Groups
"ODK Community" group.
To unsubscribe from this group and stop receiving emails from it,
send
an
email to opendatakit...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--
Mitch Sundt
Software Engineer
http://www.OpenDataKit.org
University of Washington
mitche...@gmail.com

--
Mitch Sundt
Software Engineer
http://www.OpenDataKit.org http://www.opendatakit.org/
University of Washington
mitchellsundt@gmail.com

Hi Yom,

I've filed this at
https://github.com/opendatakit/opendatakit/issues/1253. It's not a
small amount of work, so it's unlikely to be fixed soon.

If all the forms are on the enumerator's devices, I would just gather
them at the end of the campaign, pull the data off the devices with
ODK Briefcase and export to CSV.

Yaw

··· -- Need ODK consultants? Nafundi provides form design, server setup, in-field training, and software development for ODK. Go to https://nafundi.com to get started.

On Mon, Sep 12, 2016 at 5:47 PM, Mitchell Sundt msundt@cs.washington.edu wrote:

Various parts to this:

4 hr: The plumbing between the GWT-based UI (new button, new pop-up dialog)
and a servlet for this, and the changes to the web.xml and security
configuration to show everything is about 4 hours (an example would be the
upload-permissions-csv feature in 1.4.12; note: that is spread across
several check-ins).

3 days: I would push the GQL / SQL syntax construction down into the
persistence engine implementations. This is already what is done internally
with queries in the PostgreSQL / MySQL case, so you'd just need to grab the
query string without executing the query. For Google, you would need to add
that functionality (since it is all API-based). Probably about 2-3 days for
this to understand the layers, add the new APIs and write the Google GQL
syntax writer.

3-10 days: Writing the servlet that would lay out HTML content with details
on a specific instance ID within a table. It depends upon how fancy you want
to make the page and making sure you handle all the shadow, select-multiple,
repeat group, and attachment tables correctly.

On Thu, Sep 8, 2016 at 2:42 AM, Yaw Anokwa yanokwa@nafundi.com wrote:

Hi Mitch,

Do you have a sense for how much work (hours, days, weeks?) it'd be to
generate such a report?

Thanks,

Yaw

Need ODK consultants? Nafundi provides form design, server setup,
in-field training, and software development for ODK. Go to
https://nafundi.com to get started.

On Thu, Sep 8, 2016 at 4:55 AM, Guillaume Lestrelin g.lestrelin@gmail.com wrote:

Thanks for the detailed explanation Mitch.

A more comprehensive "Repair Submission" reporting system would
certainly be
useful as Aggregate currently only returns single SQL statements, i.e.
providing info on data corruption issues in only one table whereas
corrupted
data might also occur in other tables for the same submission. This
requires
the user to either (1) go back and forth between filtering a corrupted
submission in Aggregate and repairing the corrupted tables in the
Datastore
or (2) for each corrupted submission, going through every single table
in
the Datastore and repairing the corrupted data. I have combined both
options
to resolve my issue.

Finally, upgrading is definitely useful. In my case, data corruption
seems
to have occurred exclusively before I upgraded to 1.4.11 (I was
initially
using 1.4.9).

Cheers,

Yom

Le jeudi 8 septembre 2016 01:42:25 UTC+7, Mitchell Sundt a écrit :

Automating the data-corruption clean-up is problematic.

If it were to be implemented, it should not happen silently during a
publish or ODK Briefcase pull action (actions that should be guaranteed
to
be read-only and non-destructive). Those should still fail and report
the
instance ID with the data-corruption issue.

The goal should be to make altering of the data on the server an
explicit
user action. This could be via a new button on the Form Management /
Submission Admin sub-tab (e.g., "Repair Submission") that prompted for
the
instanceID of the problematic row.

But, implementing that action is tricky because of the many different
ways
that failures can occur, with duplicates or omissions of records in any
of
the form's tables leading to data corruption. While it is somewhat
straightforward in forms without repeat groups, this gets particularly
complicated when there are repeat groups because those can be
arbitrarily
deeply nested.

Given how difficult it has been to generate unique column names in the
face of large deeply-nested forms (there are still forms for which this
fails), I, personally, would not trust any code written to
automatically
repair a submission; developing tests to verify the correct handling of
all
data corruption failure cases would be similarly impossible.

Perhaps a first step would be for the "Repair Submission" action to
generate a report containing the GQL or SQL statements that users could
cut-and-paste to investigate the errors, confirm the recommendations,
and
effect the deletions themselves.

Data corruption should be reduced in likelihood by the changes in ODK
Aggregate 1.4.11 and higher.

On Mon, Aug 29, 2016 at 4:00 AM, Yaw Anokwa yan...@nafundi.com wrote:

Yom,

This is an issue that we'll have to escalate to Mitch and Waylon to
see if they have any ideas. I know they are both busy, but they will
respond when they get a moment...

Thanks,

Yaw

Need ODK consultants? Nafundi provides form design, server setup,
in-field training, and software development for ODK. Go to
https://nafundi.com to get started.

On Wed, Aug 24, 2016 at 9:18 AM, Guillaume Lestrelin g.les...@gmail.com wrote:

Yaw,

Deleting the submissions in Aggregate is not an option apparently
since
my
attempts to display / filter the problematic submissions (using
their
UUIDs)
result in error messages like: "Error: Problem persisting data or
accessing
data (SELECT * FROM
NFFARMHH_GROUP_OTHER_MEMBERS_GROUP_OTHER_GENDER_AGE
WHERE _TOP_LEVEL_AURI = uuid:46d2067c-4315-4bfc-9ec6-dd4c04027279
AND
_PARENT_AURI = uuid:46d2067c-4315-4bfc-9ec6-dd4c04027279 is missing
a
(repeat) group instance OR has an extra copies.)"

In order to test your suggestion, I deleted a few problematic
submissions
directly in the Datastore but to do so, I have to run queries and
delete
both parent and child entities (13 different tables) manually. From
what I
understand (I am not exactly a computer expert :-P), the "low-level
Java
Datastore API" does not deal with child entities when deleting
parent
entities... If I leave these child entities, I suppose this could
create new
issues when resending the forms to Aggregate, right? Any suggestions
of
a
faster/cleaner/more secure method to delete the whole 'parent-child'
groups?

Yom

Le lundi 22 août 2016 20:49:32 UTC+7, Yaw Anokwa a écrit :

Guillaume,

"This is occasionally necessary because Google can kill the server
process...These failures are more likely to occur if you are trying
to
run using just free quota, or if you are heavily using your system
and
the form being uploaded is large or contains many repeat groups."

https://github.com/opendatakit/opendatakit/wiki/Aggregate-AppEngine-Troubleshooting#repairing-your-app-engine-database-yourself

I would guess that deleting the submissions in Aggregate would
solve
the problem of errors. I would also guess that you could resend
those
forms successfully because the UUID that Aggregate uses to check
duplicates won't be on the server. I say guess because I didn't
write
that code. Please test to confirm.

Going forward the best way to prevent these errors is to run 1.4.11
and enable billing.

Yaw

Need ODK consultants? Nafundi provides form design, server setup,
in-field training, and software development for ODK. Go to
https://nafundi.com to get started.

On Mon, Aug 22, 2016 at 7:57 AM, Guillaume Lestrelin g.les...@gmail.com wrote:

Thanks Yaw,

Actually, Briefcase returns errors "fetching submission" for a
total
of
51
UUIDs (out of 1200+ so far). Looking at the Datastore, I find
duplicate
rows
for all these UUIDs in something like 10-12 tables. I wonder
about
the
underlying cause: AppEngine daily/free quotas issues? fairly slow
internet
connection in the field? Too many repeat groups in the form?
I also wonder: what about deleting the corresponding 51
submissions
(entire
completed forms/tables) directly in Aggregate or in the Datatore?
What
would
be the behaviour of ODK Collect? Would the corresponding forms be
re
sent or
not?

Cheers,

Yom

--

Post: opend...@googlegroups.com
Unsubscribe: opendatakit...@googlegroups.com
Options: http://groups.google.com/group/opendatakit?hl=en


You received this message because you are subscribed to the
Google
Groups
"ODK Community" group.
To unsubscribe from this group and stop receiving emails from it,
send
an
email to opendatakit...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--

Post: opend...@googlegroups.com
Unsubscribe: opendatakit...@googlegroups.com
Options: http://groups.google.com/group/opendatakit?hl=en


You received this message because you are subscribed to the Google
Groups
"ODK Community" group.
To unsubscribe from this group and stop receiving emails from it,
send
an
email to opendatakit...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--
Mitch Sundt
Software Engineer
http://www.OpenDataKit.org
University of Washington
mitche...@gmail.com

--
Mitch Sundt
Software Engineer
http://www.OpenDataKit.org
University of Washington
mitchellsundt@gmail.com

It's not clear how this was resolved. Are both parent and child tables
necessary and sufficient to delete?

··· On Wednesday, September 14, 2016 at 11:00:55 AM UTC-4, Yaw Anokwa wrote:

Hi Yom,

I've filed this at
https://github.com/opendatakit/opendatakit/issues/1253. It's not a
small amount of work, so it's unlikely to be fixed soon.

If all the forms are on the enumerator's devices, I would just gather
them at the end of the campaign, pull the data off the devices with
ODK Briefcase and export to CSV.

Yaw

Need ODK consultants? Nafundi provides form design, server setup,
in-field training, and software development for ODK. Go to
https://nafundi.com to get started.

On Mon, Sep 12, 2016 at 5:47 PM, Mitchell Sundt <msu...@cs.washington.edu <javascript:>> wrote:

Various parts to this:

4 hr: The plumbing between the GWT-based UI (new button, new pop-up
dialog)
and a servlet for this, and the changes to the web.xml and security
configuration to show everything is about 4 hours (an example would be
the
upload-permissions-csv feature in 1.4.12; note: that is spread across
several check-ins).

3 days: I would push the GQL / SQL syntax construction down into the
persistence engine implementations. This is already what is done
internally
with queries in the PostgreSQL / MySQL case, so you'd just need to grab
the
query string without executing the query. For Google, you would need to
add
that functionality (since it is all API-based). Probably about 2-3 days
for
this to understand the layers, add the new APIs and write the Google GQL
syntax writer.

3-10 days: Writing the servlet that would lay out HTML content with
details
on a specific instance ID within a table. It depends upon how fancy you
want
to make the page and making sure you handle all the shadow,
select-multiple,
repeat group, and attachment tables correctly.

On Thu, Sep 8, 2016 at 2:42 AM, Yaw Anokwa <yan...@nafundi.com <javascript:>> wrote:

Hi Mitch,

Do you have a sense for how much work (hours, days, weeks?) it'd be to
generate such a report?

Thanks,

Yaw

Need ODK consultants? Nafundi provides form design, server setup,
in-field training, and software development for ODK. Go to
https://nafundi.com to get started.

On Thu, Sep 8, 2016 at 4:55 AM, Guillaume Lestrelin <g.les...@gmail.com <javascript:>> wrote:

Thanks for the detailed explanation Mitch.

A more comprehensive "Repair Submission" reporting system would
certainly be
useful as Aggregate currently only returns single SQL statements,
i.e.
providing info on data corruption issues in only one table whereas
corrupted
data might also occur in other tables for the same submission. This
requires
the user to either (1) go back and forth between filtering a
corrupted
submission in Aggregate and repairing the corrupted tables in the
Datastore
or (2) for each corrupted submission, going through every single
table
in
the Datastore and repairing the corrupted data. I have combined both
options
to resolve my issue.

Finally, upgrading is definitely useful. In my case, data corruption
seems
to have occurred exclusively before I upgraded to 1.4.11 (I was
initially
using 1.4.9).

Cheers,

Yom

Le jeudi 8 septembre 2016 01:42:25 UTC+7, Mitchell Sundt a écrit :

Automating the data-corruption clean-up is problematic.

If it were to be implemented, it should not happen silently during a
publish or ODK Briefcase pull action (actions that should be
guaranteed
to
be read-only and non-destructive). Those should still fail and
report
the
instance ID with the data-corruption issue.

The goal should be to make altering of the data on the server an
explicit
user action. This could be via a new button on the Form Management /
Submission Admin sub-tab (e.g., "Repair Submission") that prompted
for
the
instanceID of the problematic row.

But, implementing that action is tricky because of the many
different
ways
that failures can occur, with duplicates or omissions of records in
any
of
the form's tables leading to data corruption. While it is somewhat
straightforward in forms without repeat groups, this gets
particularly
complicated when there are repeat groups because those can be
arbitrarily
deeply nested.

Given how difficult it has been to generate unique column names in
the
face of large deeply-nested forms (there are still forms for which
this
fails), I, personally, would not trust any code written to
automatically
repair a submission; developing tests to verify the correct handling
of
all
data corruption failure cases would be similarly impossible.

Perhaps a first step would be for the "Repair Submission" action to
generate a report containing the GQL or SQL statements that users
could
cut-and-paste to investigate the errors, confirm the
recommendations,
and
effect the deletions themselves.

Data corruption should be reduced in likelihood by the changes in
ODK
Aggregate 1.4.11 and higher.

On Mon, Aug 29, 2016 at 4:00 AM, Yaw Anokwa yan...@nafundi.com wrote:

Yom,

This is an issue that we'll have to escalate to Mitch and Waylon to
see if they have any ideas. I know they are both busy, but they
will
respond when they get a moment...

Thanks,

Yaw

Need ODK consultants? Nafundi provides form design, server setup,
in-field training, and software development for ODK. Go to
https://nafundi.com to get started.

On Wed, Aug 24, 2016 at 9:18 AM, Guillaume Lestrelin g.les...@gmail.com wrote:

Yaw,

Deleting the submissions in Aggregate is not an option apparently
since
my
attempts to display / filter the problematic submissions (using
their
UUIDs)
result in error messages like: "Error: Problem persisting data or
accessing
data (SELECT * FROM
NFFARMHH_GROUP_OTHER_MEMBERS_GROUP_OTHER_GENDER_AGE
WHERE _TOP_LEVEL_AURI = uuid:46d2067c-4315-4bfc-9ec6-dd4c04027279
AND
_PARENT_AURI = uuid:46d2067c-4315-4bfc-9ec6-dd4c04027279 is
missing
a
(repeat) group instance OR has an extra copies.)"

In order to test your suggestion, I deleted a few problematic
submissions
directly in the Datastore but to do so, I have to run queries and
delete
both parent and child entities (13 different tables) manually.
From
what I
understand (I am not exactly a computer expert :-P), the
"low-level
Java
Datastore API" does not deal with child entities when deleting
parent
entities... If I leave these child entities, I suppose this could
create new
issues when resending the forms to Aggregate, right? Any
suggestions
of
a
faster/cleaner/more secure method to delete the whole
'parent-child'
groups?

Yom

Le lundi 22 août 2016 20:49:32 UTC+7, Yaw Anokwa a écrit :

Guillaume,

"This is occasionally necessary because Google can kill the
server
process...These failures are more likely to occur if you are
trying
to
run using just free quota, or if you are heavily using your
system
and
the form being uploaded is large or contains many repeat
groups."

https://github.com/opendatakit/opendatakit/wiki/Aggregate-AppEngine-Troubleshooting#repairing-your-app-engine-database-yourself

I would guess that deleting the submissions in Aggregate would
solve
the problem of errors. I would also guess that you could resend
those
forms successfully because the UUID that Aggregate uses to check
duplicates won't be on the server. I say guess because I didn't
write
that code. Please test to confirm.

Going forward the best way to prevent these errors is to run
1.4.11
and enable billing.

Yaw

Need ODK consultants? Nafundi provides form design, server
setup,
in-field training, and software development for ODK. Go to
https://nafundi.com to get started.

On Mon, Aug 22, 2016 at 7:57 AM, Guillaume Lestrelin g.les...@gmail.com wrote:

Thanks Yaw,

Actually, Briefcase returns errors "fetching submission" for a
total
of
51
UUIDs (out of 1200+ so far). Looking at the Datastore, I find
duplicate
rows
for all these UUIDs in something like 10-12 tables. I wonder
about
the
underlying cause: AppEngine daily/free quotas issues? fairly
slow
internet
connection in the field? Too many repeat groups in the form?
I also wonder: what about deleting the corresponding 51
submissions
(entire
completed forms/tables) directly in Aggregate or in the
Datatore?
What
would
be the behaviour of ODK Collect? Would the corresponding forms
be
re
sent or
not?

Cheers,

Yom

--

Post: opend...@googlegroups.com
Unsubscribe: opendatakit...@googlegroups.com
Options: http://groups.google.com/group/opendatakit?hl=en


You received this message because you are subscribed to the
Google
Groups
"ODK Community" group.
To unsubscribe from this group and stop receiving emails from
it,
send
an
email to opendatakit...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--

Post: opend...@googlegroups.com
Unsubscribe: opendatakit...@googlegroups.com
Options: http://groups.google.com/group/opendatakit?hl=en


You received this message because you are subscribed to the
Google
Groups
"ODK Community" group.
To unsubscribe from this group and stop receiving emails from it,
send
an
email to opendatakit...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--
Mitch Sundt
Software Engineer
http://www.OpenDataKit.org
University of Washington
mitche...@gmail.com

--
Mitch Sundt
Software Engineer
http://www.OpenDataKit.org
University of Washington
mitche...@gmail.com <javascript:>

Hi Brett,

The options are either to edit the DB manually as described at

or if you are a developer, review Mitch's proposal to automate the
process at https://github.com/opendatakit/opendatakit/issues/1253 and
submit code that does that.

Thanks,

Yaw

··· On Wed, Feb 1, 2017 at 12:53 PM, Brett Gall wrote: > It's not clear how this was resolved. Are both parent and child tables > necessary and sufficient to delete? > > > On Wednesday, September 14, 2016 at 11:00:55 AM UTC-4, Yaw Anokwa wrote: >> >> Hi Yom, >> >> I've filed this at >> https://github.com/opendatakit/opendatakit/issues/1253. It's not a >> small amount of work, so it's unlikely to be fixed soon. >> >> If all the forms are on the enumerator's devices, I would just gather >> them at the end of the campaign, pull the data off the devices with >> ODK Briefcase and export to CSV. >> >> Yaw >> -- >> Need ODK consultants? Nafundi provides form design, server setup, >> in-field training, and software development for ODK. Go to >> https://nafundi.com to get started. >> >> On Mon, Sep 12, 2016 at 5:47 PM, Mitchell Sundt wrote: >> > Various parts to this: >> > >> > 4 hr: The plumbing between the GWT-based UI (new button, new pop-up >> > dialog) >> > and a servlet for this, and the changes to the web.xml and security >> > configuration to show everything is about 4 hours (an example would be >> > the >> > upload-permissions-csv feature in 1.4.12; note: that is spread across >> > several check-ins). >> > >> > 3 days: I would push the GQL / SQL syntax construction down into the >> > persistence engine implementations. This is already what is done >> > internally >> > with queries in the PostgreSQL / MySQL case, so you'd just need to grab >> > the >> > query string without executing the query. For Google, you would need to >> > add >> > that functionality (since it is all API-based). Probably about 2-3 days >> > for >> > this to understand the layers, add the new APIs and write the Google GQL >> > syntax writer. >> > >> > 3-10 days: Writing the servlet that would lay out HTML content with >> > details >> > on a specific instance ID within a table. It depends upon how fancy you >> > want >> > to make the page and making sure you handle all the shadow, >> > select-multiple, >> > repeat group, and attachment tables correctly. >> > >> > >> > >> > >> > >> > >> > On Thu, Sep 8, 2016 at 2:42 AM, Yaw Anokwa wrote: >> >> >> >> Hi Mitch, >> >> >> >> Do you have a sense for how much work (hours, days, weeks?) it'd be to >> >> generate such a report? >> >> >> >> Thanks, >> >> >> >> Yaw >> >> -- >> >> Need ODK consultants? Nafundi provides form design, server setup, >> >> in-field training, and software development for ODK. Go to >> >> https://nafundi.com to get started. >> >> >> >> On Thu, Sep 8, 2016 at 4:55 AM, Guillaume Lestrelin wrote: >> >> > Thanks for the detailed explanation Mitch. >> >> > >> >> > A more comprehensive "Repair Submission" reporting system would >> >> > certainly be >> >> > useful as Aggregate currently only returns single SQL statements, >> >> > i.e. >> >> > providing info on data corruption issues in only one table whereas >> >> > corrupted >> >> > data might also occur in other tables for the same submission. This >> >> > requires >> >> > the user to either (1) go back and forth between filtering a >> >> > corrupted >> >> > submission in Aggregate and repairing the corrupted tables in the >> >> > Datastore >> >> > or (2) for each corrupted submission, going through every single >> >> > table >> >> > in >> >> > the Datastore and repairing the corrupted data. I have combined both >> >> > options >> >> > to resolve my issue. >> >> > >> >> > Finally, upgrading is definitely useful. In my case, data corruption >> >> > seems >> >> > to have occurred exclusively before I upgraded to 1.4.11 (I was >> >> > initially >> >> > using 1.4.9). >> >> > >> >> > Cheers, >> >> > >> >> > Yom >> >> > >> >> > >> >> > Le jeudi 8 septembre 2016 01:42:25 UTC+7, Mitchell Sundt a écrit : >> >> >> >> >> >> Automating the data-corruption clean-up is problematic. >> >> >> >> >> >> If it were to be implemented, it should not happen silently during a >> >> >> publish or ODK Briefcase pull action (actions that should be >> >> >> guaranteed >> >> >> to >> >> >> be read-only and non-destructive). Those should still fail and >> >> >> report >> >> >> the >> >> >> instance ID with the data-corruption issue. >> >> >> >> >> >> The goal should be to make altering of the data on the server an >> >> >> explicit >> >> >> user action. This could be via a new button on the Form Management / >> >> >> Submission Admin sub-tab (e.g., "Repair Submission") that prompted >> >> >> for >> >> >> the >> >> >> instanceID of the problematic row. >> >> >> >> >> >> But, implementing that action is tricky because of the many >> >> >> different >> >> >> ways >> >> >> that failures can occur, with duplicates or omissions of records in >> >> >> any >> >> >> of >> >> >> the form's tables leading to data corruption. While it is somewhat >> >> >> straightforward in forms without repeat groups, this gets >> >> >> particularly >> >> >> complicated when there are repeat groups because those can be >> >> >> arbitrarily >> >> >> deeply nested. >> >> >> >> >> >> Given how difficult it has been to generate unique column names in >> >> >> the >> >> >> face of large deeply-nested forms (there are still forms for which >> >> >> this >> >> >> fails), I, personally, would not trust any code written to >> >> >> automatically >> >> >> repair a submission; developing tests to verify the correct handling >> >> >> of >> >> >> all >> >> >> data corruption failure cases would be similarly impossible. >> >> >> >> >> >> Perhaps a first step would be for the "Repair Submission" action to >> >> >> generate a report containing the GQL or SQL statements that users >> >> >> could >> >> >> cut-and-paste to investigate the errors, confirm the >> >> >> recommendations, >> >> >> and >> >> >> effect the deletions themselves. >> >> >> >> >> >> Data corruption should be reduced in likelihood by the changes in >> >> >> ODK >> >> >> Aggregate 1.4.11 and higher. >> >> >> >> >> >> >> >> >> On Mon, Aug 29, 2016 at 4:00 AM, Yaw Anokwa wrote: >> >> >>> >> >> >>> Yom, >> >> >>> >> >> >>> This is an issue that we'll have to escalate to Mitch and Waylon to >> >> >>> see if they have any ideas. I know they are both busy, but they >> >> >>> will >> >> >>> respond when they get a moment... >> >> >>> >> >> >>> Thanks, >> >> >>> >> >> >>> Yaw >> >> >>> -- >> >> >>> Need ODK consultants? Nafundi provides form design, server setup, >> >> >>> in-field training, and software development for ODK. Go to >> >> >>> https://nafundi.com to get started. >> >> >>> >> >> >>> On Wed, Aug 24, 2016 at 9:18 AM, Guillaume Lestrelin wrote: >> >> >>> > Yaw, >> >> >>> > >> >> >>> > Deleting the submissions in Aggregate is not an option apparently >> >> >>> > since >> >> >>> > my >> >> >>> > attempts to display / filter the problematic submissions (using >> >> >>> > their >> >> >>> > UUIDs) >> >> >>> > result in error messages like: "Error: Problem persisting data or >> >> >>> > accessing >> >> >>> > data (SELECT * FROM >> >> >>> > NFFARMHH_GROUP_OTHER_MEMBERS_GROUP_OTHER_GENDER_AGE >> >> >>> > WHERE _TOP_LEVEL_AURI = uuid:46d2067c-4315-4bfc-9ec6-dd4c04027279 >> >> >>> > AND >> >> >>> > _PARENT_AURI = uuid:46d2067c-4315-4bfc-9ec6-dd4c04027279 is >> >> >>> > missing >> >> >>> > a >> >> >>> > (repeat) group instance OR has an extra copies.)" >> >> >>> > >> >> >>> > In order to test your suggestion, I deleted a few problematic >> >> >>> > submissions >> >> >>> > directly in the Datastore but to do so, I have to run queries and >> >> >>> > delete >> >> >>> > both parent and child entities (13 different tables) manually. >> >> >>> > From >> >> >>> > what I >> >> >>> > understand (I am not exactly a computer expert :-P), the >> >> >>> > "low-level >> >> >>> > Java >> >> >>> > Datastore API" does not deal with child entities when deleting >> >> >>> > parent >> >> >>> > entities... If I leave these child entities, I suppose this could >> >> >>> > create new >> >> >>> > issues when resending the forms to Aggregate, right? Any >> >> >>> > suggestions >> >> >>> > of >> >> >>> > a >> >> >>> > faster/cleaner/more secure method to delete the whole >> >> >>> > 'parent-child' >> >> >>> > groups? >> >> >>> > >> >> >>> > Yom >> >> >>> > >> >> >>> > >> >> >>> > Le lundi 22 août 2016 20:49:32 UTC+7, Yaw Anokwa a écrit : >> >> >>> >> >> >> >>> >> Guillaume, >> >> >>> >> >> >> >>> >> "This is occasionally necessary because Google can kill the >> >> >>> >> server >> >> >>> >> process...These failures are more likely to occur if you are >> >> >>> >> trying >> >> >>> >> to >> >> >>> >> run using just free quota, or if you are heavily using your >> >> >>> >> system >> >> >>> >> and >> >> >>> >> the form being uploaded is large or contains many repeat >> >> >>> >> groups." >> >> >>> >> >> >> >>> >> >> >> >>> >> >> >> >>> >> >> >> >>> >> https://github.com/opendatakit/opendatakit/wiki/Aggregate-AppEngine-Troubleshooting#repairing-your-app-engine-database-yourself >> >> >>> >> >> >> >>> >> I would guess that deleting the submissions in Aggregate would >> >> >>> >> solve >> >> >>> >> the problem of errors. I would also guess that you could resend >> >> >>> >> those >> >> >>> >> forms successfully because the UUID that Aggregate uses to check >> >> >>> >> duplicates won't be on the server. I say guess because I didn't >> >> >>> >> write >> >> >>> >> that code. Please test to confirm. >> >> >>> >> >> >> >>> >> Going forward the best way to prevent these errors is to run >> >> >>> >> 1.4.11 >> >> >>> >> and enable billing. >> >> >>> >> >> >> >>> >> Yaw >> >> >>> >> -- >> >> >>> >> Need ODK consultants? Nafundi provides form design, server >> >> >>> >> setup, >> >> >>> >> in-field training, and software development for ODK. Go to >> >> >>> >> https://nafundi.com to get started. >> >> >>> >> >> >> >>> >> On Mon, Aug 22, 2016 at 7:57 AM, Guillaume Lestrelin wrote: >> >> >>> >> > Thanks Yaw, >> >> >>> >> > >> >> >>> >> > Actually, Briefcase returns errors "fetching submission" for a >> >> >>> >> > total >> >> >>> >> > of >> >> >>> >> > 51 >> >> >>> >> > UUIDs (out of 1200+ so far). Looking at the Datastore, I find >> >> >>> >> > duplicate >> >> >>> >> > rows >> >> >>> >> > for all these UUIDs in something like 10-12 tables. I wonder >> >> >>> >> > about >> >> >>> >> > the >> >> >>> >> > underlying cause: AppEngine daily/free quotas issues? fairly >> >> >>> >> > slow >> >> >>> >> > internet >> >> >>> >> > connection in the field? Too many repeat groups in the form? >> >> >>> >> > I also wonder: what about deleting the corresponding 51 >> >> >>> >> > submissions >> >> >>> >> > (entire >> >> >>> >> > completed forms/tables) directly in Aggregate or in the >> >> >>> >> > Datatore? >> >> >>> >> > What >> >> >>> >> > would >> >> >>> >> > be the behaviour of ODK Collect? Would the corresponding forms >> >> >>> >> > be >> >> >>> >> > re >> >> >>> >> > sent or >> >> >>> >> > not? >> >> >>> >> > >> >> >>> >> > Cheers, >> >> >>> >> > >> >> >>> >> > Yom >> >> >>> >> > >> >> >>> >> > -- >> >> >>> >> > -- >> >> >>> >> > Post: opend...@googlegroups.com >> >> >>> >> > Unsubscribe: opendatakit...@googlegroups.com >> >> >>> >> > Options: http://groups.google.com/group/opendatakit?hl=en >> >> >>> >> > >> >> >>> >> > --- >> >> >>> >> > You received this message because you are subscribed to the >> >> >>> >> > Google >> >> >>> >> > Groups >> >> >>> >> > "ODK Community" group. >> >> >>> >> > To unsubscribe from this group and stop receiving emails from >> >> >>> >> > it, >> >> >>> >> > send >> >> >>> >> > an >> >> >>> >> > email to opendatakit...@googlegroups.com. >> >> >>> >> > For more options, visit https://groups.google.com/d/optout. >> >> >>> > >> >> >>> > -- >> >> >>> > -- >> >> >>> > Post: opend...@googlegroups.com >> >> >>> > Unsubscribe: opendatakit...@googlegroups.com >> >> >>> > Options: http://groups.google.com/group/opendatakit?hl=en >> >> >>> > >> >> >>> > --- >> >> >>> > You received this message because you are subscribed to the >> >> >>> > Google >> >> >>> > Groups >> >> >>> > "ODK Community" group. >> >> >>> > To unsubscribe from this group and stop receiving emails from it, >> >> >>> > send >> >> >>> > an >> >> >>> > email to opendatakit...@googlegroups.com. >> >> >>> > For more options, visit https://groups.google.com/d/optout. >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> -- >> >> >> Mitch Sundt >> >> >> Software Engineer >> >> >> http://www.OpenDataKit.org >> >> >> University of Washington >> >> >> mitche...@gmail.com >> >> >> >> >> > >> > >> > >> > >> > >> > -- >> > Mitch Sundt >> > Software Engineer >> > http://www.OpenDataKit.org >> > University of Washington >> > mitche...@gmail.com >> > > > -- > -- > Post: opendatakit@googlegroups.com > Unsubscribe: opendatakit+unsubscribe@googlegroups.com > Options: http://groups.google.com/group/opendatakit?hl=en > > --- > You received this message because you are subscribed to the Google Groups > "ODK Community" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to opendatakit+unsubscribe@googlegroups.com. > > For more options, visit https://groups.google.com/d/optout.