Error on form upload to Aggregate

Colin · June 8, 2015, 3:27pm

Hi all,

I am attempting to upload a rather large XML form to ODK Aggregate (about 27000 lines). I am getting the following error:

HTTP Status 400 - Problem parsing submission XML org.opendatakit.aggregate.exception.ODKParseException: Error processing new form: org.opendatakit.aggregate.exception.ODKParseException: Unable to create form data tables after 100 attempts.

I know this is not an obvious parse error in the sense that the code is valid (has been validated).

Looking into the source code, it seems that this error would be thrown during the "Create the backing tables" step. Despite the "parse error" terminology, I wonder if it has to do with the size of the file? Maybe I need to increase the MAX_FORM_CREATION_ATTEMPTS to 500 or something? Hopefully there is an easier way... any suggestions?

Thanks,
Colin

Mitch_S · June 8, 2015, 7:23pm

First, are you using the latest ODK Aggregate?

And are you uploading to AppEngine, or running a Tomcat server against
MySQL or PostgreSQL?

If your form is very large, the data needs to be split across multiple
tables. The failure is likely because of that splitting process is failing.

On MySQL, there is a limit to how big a row of data can be. You cannot
specify a string field length greater than about 16000 (

)

If you are using ODK Aggregate 1.4.7, and are having problems, you can send
me your form and I will see what the error is.

The easy work-around is often just renaming your group names and/or field
names.

Keeping your field names and group names short and making them as different
from each other as possible helps (e.g., 'user_details_context_home',
'user_details_context_work' for groups is not very good. Better names might
be 'home_details', 'work_details').

···

On Mon, Jun 8, 2015 at 8:27 AM, wrote:

Hi all,

I am attempting to upload a rather large XML form to ODK Aggregate (about
27000 lines). I am getting the following error:

HTTP Status 400 - Problem parsing submission XML
org.opendatakit.aggregate.exception.ODKParseException: Error processing new
form: org.opendatakit.aggregate.exception.ODKParseException: Unable to
create form data tables after 100 attempts.

I know this is not an obvious parse error in the sense that the code is
valid (has been validated).

Looking into the source code, it seems that this error would be thrown
during the "Create the backing tables" step. Despite the "parse error"
terminology, I wonder if it has to do with the size of the file? Maybe I
need to increase the MAX_FORM_CREATION_ATTEMPTS to 500 or something?
Hopefully there is an easier way... any suggestions?

Thanks,
Colin

--

Post: opendatakit@googlegroups.com
Unsubscribe: opendatakit+unsubscribe@googlegroups.com
Options: http://groups.google.com/group/opendatakit?hl=en

You received this message because you are subscribed to the Google Groups
"ODK Community" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to opendatakit+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--
Mitch Sundt
Software Engineer
University of Washington
mitchellsundt@gmail.com

Colin · June 9, 2015, 8:18pm

Thanks Mitch,

Unfortunately, we're on a slightly older version of Aggregate (1.3.1) and now is not a good time to upgrade (data collection ongoing)... but if we knew this would solve the problem I could probably convince people. We are running Tomcat against MySQL.

The form is, I believe, very large. Certainly the largest we've dealt with (27400 lines of nicely formatted xml, if that's a useful metric, or 1.4 mb).

I ran some test to see if I could narrow things down.
If I delete half of the form, getting the size down to about 600 kb, it uploads.
I tried shortening the field names as much as possible (while still conveying some vaguely humane readable information), but was apparently not able to get them short enough.

I assume this is the kind of thing that would be causing the issue:

<EMRITE10_gp>
<EMRITE10_PRADT/>
<EMRITE10_NP_gp>
<EMRITE10_NP/>
<EMRITE10_NP_LF/>
</EMRITE10_NP_gp>
<EMRITE10_VOL/>
</EMRITE10_gp>

<EMRITE11_gp>
<EMRITE11_PRADT/>
<EMRITE11_NP_gp>
<EMRITE11_NP/>
<EMRITE11_NP_LF/>
</EMRITE11_NP_gp>
<EMRITE11_VOL/>
</EMRITE11_gp>

<EMRITE12_gp>
<EMRITE12_PRADT/>
<EMRITE12_NP_gp>
<EMRITE12_NP/>
<EMRITE12_NP_LF/>
</EMRITE12_NP_gp>
<EMRITE12_VOL/>
</EMRITE12_gp>

So you end up with paths that look like:
/data/EMRITE1_gp/EMRITE1_NP_gp/EMRITE1_NP_LF:option0

There are 29 medications (of which EMRITE is one), and 12 data entry points for each (I know there are cleaner ways to do this with looping but the data analyst wasn't having it). Hence the size.

So I gather from the docs that and your comments that this type of repetition of field names is likely to be causing the problem? I'm not sure how to shorten them any further... I guess the fall back is to split the form into two, but any other suggestions would be welcome.

Thanks again,
Colin

···

On Monday, June 8, 2015 at 3:23:49 PM UTC-4, Mitch Sundt wrote: > First, are you using the latest ODK Aggregate? > > And are you uploading to AppEngine, or running a Tomcat server against MySQL or PostgreSQL? > > If your form is very large, the data needs to be split across multiple tables. The failure is likely because of that splitting process is failing. > > > > On MySQL, there is a limit to how big a row of data can be. You cannot specify a string field length greater than about 16000 ( https://opendatakit.org/help/form-design/guidelines/#datastore_string_length ) > > > > If you are using ODK Aggregate 1.4.7, and are having problems, you can send me your form and I will see what the error is. > > The easy work-around is often just renaming your group names and/or field names. > > > Keeping your field names and group names short and making them as different from each other as possible helps (e.g., 'user_details_context_home', 'user_details_context_work' for groups is not very good. Better names might be 'home_details', 'work_details'). > > > > > > > On Mon, Jun 8, 2015 at 8:27 AM, wrote: > Hi all, > > > > I am attempting to upload a rather large XML form to ODK Aggregate (about 27000 lines). I am getting the following error: > > > > HTTP Status 400 - Problem parsing submission XML org.opendatakit.aggregate.exception.ODKParseException: Error processing new form: org.opendatakit.aggregate.exception.ODKParseException: Unable to create form data tables after 100 attempts. > > > > I know this is not an obvious parse error in the sense that the code is valid (has been validated). > > > > Looking into the source code, it seems that this error would be thrown during the "Create the backing tables" step. Despite the "parse error" terminology, I wonder if it has to do with the size of the file? Maybe I need to increase the MAX_FORM_CREATION_ATTEMPTS to 500 or something? Hopefully there is an easier way... any suggestions? > > > > Thanks, > > Colin > > > > -- > > -- > > Post: opend...@googlegroups.com > > Unsubscribe: opendatakit...@googlegroups.com > > Options: http://groups.google.com/group/opendatakit?hl=en > > > > --- > > You received this message because you are subscribed to the Google Groups "ODK Community" group. > > To unsubscribe from this group and stop receiving emails from it, send an email to opendatakit...@googlegroups.com. > > For more options, visit https://groups.google.com/d/optout. > > > > > -- > > Mitch Sundt > Software Engineer > University of Washington > mitche...@gmail.com

Mitch_S · June 9, 2015, 9:04pm

It is highly likely that upgrading will fix your problem. There are
definitely issues with 1.3.1. The last fix to this issue was in 1.4.4. The
only way to tell is to set up a 1.4.7 server for testing purposes, and
confirm that you can load the form into it.

The ODK tools are designed for disconnected operation. So it should not be
a huge problem to take the server offline for the time needed to do a
backup and perform these updates.

The upgrade steps are outlined in the release notes.

https://code.google.com/p/opendatakit/wiki/AggregateReleaseNotes

I recommend upgrading to the next version that had a manual upgrade step,
confirming that that upgrade goes OK, then upgrading to the next, etc.

···

On Tue, Jun 9, 2015 at 1:18 PM, wrote:

Thanks Mitch,

Unfortunately, we're on a slightly older version of Aggregate (1.3.1) and
now is not a good time to upgrade (data collection ongoing)... but if we
knew this would solve the problem I could probably convince people. We are
running Tomcat against MySQL.

The form is, I believe, very large. Certainly the largest we've dealt with
(27400 lines of nicely formatted xml, if that's a useful metric, or 1.4 mb).

I ran some test to see if I could narrow things down.
If I delete half of the form, getting the size down to about 600 kb, it
uploads.
I tried shortening the field names as much as possible (while still
conveying some vaguely humane readable information), but was apparently not
able to get them short enough.

I assume this is the kind of thing that would be causing the issue:

<EMRITE10_gp>
<EMRITE10_PRADT/>
<EMRITE10_NP_gp>
<EMRITE10_NP/>
<EMRITE10_NP_LF/>
</EMRITE10_NP_gp>
<EMRITE10_VOL/>
</EMRITE10_gp>

<EMRITE11_gp>
<EMRITE11_PRADT/>
<EMRITE11_NP_gp>
<EMRITE11_NP/>
<EMRITE11_NP_LF/>
</EMRITE11_NP_gp>
<EMRITE11_VOL/>
</EMRITE11_gp>

<EMRITE12_gp>
<EMRITE12_PRADT/>
<EMRITE12_NP_gp>
<EMRITE12_NP/>
<EMRITE12_NP_LF/>
</EMRITE12_NP_gp>
<EMRITE12_VOL/>
</EMRITE12_gp>

So you end up with paths that look like:
/data/EMRITE1_gp/EMRITE1_NP_gp/EMRITE1_NP_LF:option0

There are 29 medications (of which EMRITE is one), and 12 data entry
points for each (I know there are cleaner ways to do this with looping but
the data analyst wasn't having it). Hence the size.

So I gather from the docs that and your comments that this type of
repetition of field names is likely to be causing the problem? I'm not sure
how to shorten them any further... I guess the fall back is to split the
form into two, but any other suggestions would be welcome.

Thanks again,
Colin

On Monday, June 8, 2015 at 3:23:49 PM UTC-4, Mitch Sundt wrote:

First, are you using the latest ODK Aggregate?

And are you uploading to AppEngine, or running a Tomcat server against
MySQL or PostgreSQL?

If your form is very large, the data needs to be split across multiple
tables. The failure is likely because of that splitting process is failing.

On MySQL, there is a limit to how big a row of data can be. You cannot
specify a string field length greater than about 16000 (
https://opendatakit.org/help/form-design/guidelines/#datastore_string_length
)

If you are using ODK Aggregate 1.4.7, and are having problems, you can
send me your form and I will see what the error is.

The easy work-around is often just renaming your group names and/or
field names.

Keeping your field names and group names short and making them as
different from each other as possible helps (e.g.,
'user_details_context_home', 'user_details_context_work' for groups is not
very good. Better names might be 'home_details', 'work_details').

On Mon, Jun 8, 2015 at 8:27 AM, colind...@gmail.com wrote:
Hi all,

I am attempting to upload a rather large XML form to ODK Aggregate
(about 27000 lines). I am getting the following error:

HTTP Status 400 - Problem parsing submission XML
org.opendatakit.aggregate.exception.ODKParseException: Error processing new
form: org.opendatakit.aggregate.exception.ODKParseException: Unable to
create form data tables after 100 attempts.

I know this is not an obvious parse error in the sense that the code is
valid (has been validated).

Looking into the source code, it seems that this error would be thrown
during the "Create the backing tables" step. Despite the "parse error"
terminology, I wonder if it has to do with the size of the file? Maybe I
need to increase the MAX_FORM_CREATION_ATTEMPTS to 500 or something?
Hopefully there is an easier way... any suggestions?

Thanks,

Colin

--

--

Post: opend...@googlegroups.com

Unsubscribe: opendatakit...@googlegroups.com

Options: http://groups.google.com/group/opendatakit?hl=en

You received this message because you are subscribed to the Google
Groups "ODK Community" group.

To unsubscribe from this group and stop receiving emails from it, send
an email to opendatakit...@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--

Mitch Sundt
Software Engineer
University of Washington
mitche...@gmail.com

--

Post: opendatakit@googlegroups.com
Unsubscribe: opendatakit+unsubscribe@googlegroups.com
Options: http://groups.google.com/group/opendatakit?hl=en

You received this message because you are subscribed to the Google Groups
"ODK Community" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to opendatakit+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--
Mitch Sundt
Software Engineer
University of Washington
mitchellsundt@gmail.com

Colin · June 10, 2015, 8:47pm

Ok, thanks Mitch. I'll run it upstairs and hope they agree...

···

On Tuesday, June 9, 2015 at 5:04:58 PM UTC-4, Mitch Sundt wrote: > It is highly likely that upgrading will fix your problem. There are definitely issues with 1.3.1. The last fix to this issue was in 1.4.4. The only way to tell is to set up a 1.4.7 server for testing purposes, and confirm that you can load the form into it. > > > The ODK tools are designed for disconnected operation. So it should not be a huge problem to take the server offline for the time needed to do a backup and perform these updates. > > The upgrade steps are outlined in the release notes. > > https://code.google.com/p/opendatakit/wiki/AggregateReleaseNotes > > > I recommend upgrading to the next version that had a manual upgrade step, confirming that that upgrade goes OK, then upgrading to the next, etc. > > > > > On Tue, Jun 9, 2015 at 1:18 PM, wrote: > Thanks Mitch, > > > > Unfortunately, we're on a slightly older version of Aggregate (1.3.1) and now is not a good time to upgrade (data collection ongoing)... but if we knew this would solve the problem I could probably convince people. We are running Tomcat against MySQL. > > > > The form is, I believe, very large. Certainly the largest we've dealt with (27400 lines of nicely formatted xml, if that's a useful metric, or 1.4 mb). > > > > I ran some test to see if I could narrow things down. > > If I delete half of the form, getting the size down to about 600 kb, it uploads. > > I tried shortening the field names as much as possible (while still conveying some vaguely humane readable information), but was apparently not able to get them short enough. > > > > I assume this is the kind of thing that would be causing the issue: > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > So you end up with paths that look like: > > /data/EMRITE1_gp/EMRITE1_NP_gp/EMRITE1_NP_LF:option0 > > > > There are 29 medications (of which EMRITE is one), and 12 data entry points for each (I know there are cleaner ways to do this with looping but the data analyst wasn't having it). Hence the size. > > > > So I gather from the docs that and your comments that this type of repetition of field names is likely to be causing the problem? I'm not sure how to shorten them any further... I guess the fall back is to split the form into two, but any other suggestions would be welcome. > > > > > > Thanks again, > > Colin > > > > > > > > On Monday, June 8, 2015 at 3:23:49 PM UTC-4, Mitch Sundt wrote: > > > First, are you using the latest ODK Aggregate? > > > > > > And are you uploading to AppEngine, or running a Tomcat server against MySQL or PostgreSQL? > > > > > > If your form is very large, the data needs to be split across multiple tables. The failure is likely because of that splitting process is failing. > > > > > > > > > > > > On MySQL, there is a limit to how big a row of data can be. You cannot specify a string field length greater than about 16000 ( https://opendatakit.org/help/form-design/guidelines/#datastore_string_length ) > > > > > > > > > > > > If you are using ODK Aggregate 1.4.7, and are having problems, you can send me your form and I will see what the error is. > > > > > > The easy work-around is often just renaming your group names and/or field names. > > > > > > > > > Keeping your field names and group names short and making them as different from each other as possible helps (e.g., 'user_details_context_home', 'user_details_context_work' for groups is not very good. Better names might be 'home_details', 'work_details'). > > > > > > > > > > > > > > > > > > > > > On Mon, Jun 8, 2015 at 8:27 AM, wrote: > > > Hi all, > > > > > > > > > > > > I am attempting to upload a rather large XML form to ODK Aggregate (about 27000 lines). I am getting the following error: > > > > > > > > > > > > HTTP Status 400 - Problem parsing submission XML org.opendatakit.aggregate.exception.ODKParseException: Error processing new form: org.opendatakit.aggregate.exception.ODKParseException: Unable to create form data tables after 100 attempts. > > > > > > > > > > > > I know this is not an obvious parse error in the sense that the code is valid (has been validated). > > > > > > > > > > > > Looking into the source code, it seems that this error would be thrown during the "Create the backing tables" step. Despite the "parse error" terminology, I wonder if it has to do with the size of the file? Maybe I need to increase the MAX_FORM_CREATION_ATTEMPTS to 500 or something? Hopefully there is an easier way... any suggestions? > > > > > > > > > > > > Thanks, > > > > > > Colin > > > > > > > > > > > > -- > > > > > > -- > > > > > > Post: opend...@googlegroups.com > > > > > > Unsubscribe: opendatakit...@googlegroups.com > > > > > > Options: http://groups.google.com/group/opendatakit?hl=en > > > > > > > > > > > > --- > > > > > > You received this message because you are subscribed to the Google Groups "ODK Community" group. > > > > > > To unsubscribe from this group and stop receiving emails from it, send an email to opendatakit...@googlegroups.com. > > > > > > For more options, visit https://groups.google.com/d/optout. > > > > > > > > > > > > > > > -- > > > > > > Mitch Sundt > > > Software Engineer > > > University of Washington > > > mitche...@gmail.com > > > > > > -- > > -- > > Post: opend...@googlegroups.com > > Unsubscribe: opendatakit...@googlegroups.com > > Options: http://groups.google.com/group/opendatakit?hl=en > > > > --- > > You received this message because you are subscribed to the Google Groups "ODK Community" group. > > To unsubscribe from this group and stop receiving emails from it, send an email to opendatakit...@googlegroups.com. > > For more options, visit https://groups.google.com/d/optout. > > > > > -- > > Mitch Sundt > Software Engineer > University of Washington > mitche...@gmail.com

Colin · August 20, 2015, 4:08pm

Just to follow up on this - I followed Mitch's suggestion, spun up a fresh instance with a newer Java and the newest Aggregate (only took two months to get the approval chain for that, sigh). This solved the problem - the large form is now uploading correctly (will continue to test, but I think we're good to go).

Thanks again!

···

On Wednesday, June 10, 2015 at 4:47:34 PM UTC-4, colind...@gmail.com wrote: > Ok, thanks Mitch. I'll run it upstairs and hope they agree... > > > > On Tuesday, June 9, 2015 at 5:04:58 PM UTC-4, Mitch Sundt wrote: > > It is highly likely that upgrading will fix your problem. There are definitely issues with 1.3.1. The last fix to this issue was in 1.4.4. The only way to tell is to set up a 1.4.7 server for testing purposes, and confirm that you can load the form into it. > > > > > > The ODK tools are designed for disconnected operation. So it should not be a huge problem to take the server offline for the time needed to do a backup and perform these updates. > > > > The upgrade steps are outlined in the release notes. > > > > https://code.google.com/p/opendatakit/wiki/AggregateReleaseNotes > > > > > > I recommend upgrading to the next version that had a manual upgrade step, confirming that that upgrade goes OK, then upgrading to the next, etc. > > > > > > > > > > On Tue, Jun 9, 2015 at 1:18 PM, wrote: > > Thanks Mitch, > > > > > > > > Unfortunately, we're on a slightly older version of Aggregate (1.3.1) and now is not a good time to upgrade (data collection ongoing)... but if we knew this would solve the problem I could probably convince people. We are running Tomcat against MySQL. > > > > > > > > The form is, I believe, very large. Certainly the largest we've dealt with (27400 lines of nicely formatted xml, if that's a useful metric, or 1.4 mb). > > > > > > > > I ran some test to see if I could narrow things down. > > > > If I delete half of the form, getting the size down to about 600 kb, it uploads. > > > > I tried shortening the field names as much as possible (while still conveying some vaguely humane readable information), but was apparently not able to get them short enough. > > > > > > > > I assume this is the kind of thing that would be causing the issue: > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > So you end up with paths that look like: > > > > /data/EMRITE1_gp/EMRITE1_NP_gp/EMRITE1_NP_LF:option0 > > > > > > > > There are 29 medications (of which EMRITE is one), and 12 data entry points for each (I know there are cleaner ways to do this with looping but the data analyst wasn't having it). Hence the size. > > > > > > > > So I gather from the docs that and your comments that this type of repetition of field names is likely to be causing the problem? I'm not sure how to shorten them any further... I guess the fall back is to split the form into two, but any other suggestions would be welcome. > > > > > > > > > > > > Thanks again, > > > > Colin > > > > > > > > > > > > > > > > On Monday, June 8, 2015 at 3:23:49 PM UTC-4, Mitch Sundt wrote: > > > > > First, are you using the latest ODK Aggregate? > > > > > > > > > > And are you uploading to AppEngine, or running a Tomcat server against MySQL or PostgreSQL? > > > > > > > > > > If your form is very large, the data needs to be split across multiple tables. The failure is likely because of that splitting process is failing. > > > > > > > > > > > > > > > > > > > > On MySQL, there is a limit to how big a row of data can be. You cannot specify a string field length greater than about 16000 ( https://opendatakit.org/help/form-design/guidelines/#datastore_string_length ) > > > > > > > > > > > > > > > > > > > > If you are using ODK Aggregate 1.4.7, and are having problems, you can send me your form and I will see what the error is. > > > > > > > > > > The easy work-around is often just renaming your group names and/or field names. > > > > > > > > > > > > > > > Keeping your field names and group names short and making them as different from each other as possible helps (e.g., 'user_details_context_home', 'user_details_context_work' for groups is not very good. Better names might be 'home_details', 'work_details'). > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > On Mon, Jun 8, 2015 at 8:27 AM, wrote: > > > > > Hi all, > > > > > > > > > > > > > > > > > > > > I am attempting to upload a rather large XML form to ODK Aggregate (about 27000 lines). I am getting the following error: > > > > > > > > > > > > > > > > > > > > HTTP Status 400 - Problem parsing submission XML org.opendatakit.aggregate.exception.ODKParseException: Error processing new form: org.opendatakit.aggregate.exception.ODKParseException: Unable to create form data tables after 100 attempts. > > > > > > > > > > > > > > > > > > > > I know this is not an obvious parse error in the sense that the code is valid (has been validated). > > > > > > > > > > > > > > > > > > > > Looking into the source code, it seems that this error would be thrown during the "Create the backing tables" step. Despite the "parse error" terminology, I wonder if it has to do with the size of the file? Maybe I need to increase the MAX_FORM_CREATION_ATTEMPTS to 500 or something? Hopefully there is an easier way... any suggestions? > > > > > > > > > > > > > > > > > > > > Thanks, > > > > > > > > > > Colin > > > > > > > > > > > > > > > > > > > > -- > > > > > > > > > > -- > > > > > > > > > > Post: opend...@googlegroups.com > > > > > > > > > > Unsubscribe: opendatakit...@googlegroups.com > > > > > > > > > > Options: http://groups.google.com/group/opendatakit?hl=en > > > > > > > > > > > > > > > > > > > > --- > > > > > > > > > > You received this message because you are subscribed to the Google Groups "ODK Community" group. > > > > > > > > > > To unsubscribe from this group and stop receiving emails from it, send an email to opendatakit...@googlegroups.com. > > > > > > > > > > For more options, visit https://groups.google.com/d/optout. > > > > > > > > > > > > > > > > > > > > > > > > > -- > > > > > > > > > > Mitch Sundt > > > > > Software Engineer > > > > > University of Washington > > > > > mitche...@gmail.com > > > > > > > > > > > > -- > > > > -- > > > > Post: opend...@googlegroups.com > > > > Unsubscribe: opendatakit...@googlegroups.com > > > > Options: http://groups.google.com/group/opendatakit?hl=en > > > > > > > > --- > > > > You received this message because you are subscribed to the Google Groups "ODK Community" group. > > > > To unsubscribe from this group and stop receiving emails from it, send an email to opendatakit...@googlegroups.com. > > > > For more options, visit https://groups.google.com/d/optout. > > > > > > > > > > -- > > > > Mitch Sundt > > Software Engineer > > University of Washington > > mitche...@gmail.com

Colin · August 28, 2015, 7:29pm

Need to reopen this one, unfortunately.

We have been unable to upload the large form with our new server stack. Details:
Aggregate 1.4.7
Tomcat 6
mysql Ver 14.14 Distrib 5.6.25
Java 8 (openjdk version "1.8.0_45-internal")
Ubuntu 15.04

The form definition that we're trying to upload is 1.4 MB (about 27400 lines). I can send you the form if that would be helpful.

Error as observed on Aggregate:
HTTP Status 400 - Problem parsing submission XML org.opendatakit.aggregate.exception.ODKParseException: Error processing new form: org.opendatakit.aggregate.exception.ODKParseException: Unable to create form data tables after 100 attempts.

type Status report

message Problem parsing submission XML org.opendatakit.aggregate.exception.ODKParseException: Error processing new form: org.opendatakit.aggregate.exception.ODKParseException: Unable to create form data tables after 100 attempts.

description The request sent by the client was syntactically incorrect.

Apache Tomcat/6.0.41

catalina.out logfile

INFO: Executing DROP TABLE fresh.BUILD_35524174_CORE by user anonymousUser
Aug 28, 2015 8:56:48 PM org.opendatakit.common.persistence.engine.mysql.DatastoreImpl dropRelation
WARNING: BUILD_35524174_CORE exception: org.springframework.jdbc.BadSqlGrammarException: StatementCallback; bad SQL grammar [DROP TABLE fresh.BUILD_35524174_CORE]; nested exception is com.mysql.jdbc.exceptions.MySQLSyntaxErrorException: Unknown table 'fresh.BUILD_35524174_CORE'
Aug 28, 2015 8:56:48 PM org.opendatakit.common.persistence.engine.mysql.DatastoreImpl assertRelation
INFO: before updateRelation: BUILD_35524174_CORE100
Aug 28, 2015 8:56:48 PM org.opendatakit.common.persistence.engine.mysql.DatastoreImpl assertRelation
INFO: Attempting: CREATE TABLE fresh.BUILD_35524174_CORE100 ( _URI VARCHAR(80) CHARACTER SET utf8 NOT NULL , _CREATOR_URI_USER VARCHAR(80) CHARACTER SET utf8 NOT NULL , _CREATION_DATE DATETIME NOT NULL , _LAST_UPDATE_URI_USER VARCHAR(80) CHARACTER SET utf8 NULL , _LAST_UPDATE_DATE DATETIME NOT NULL , _PARENT_AURI VARCHAR(80) CHARACTER SET utf8 NULL , _ORDINAL_NUMBER INTEGER(9 ) NOT NULL , _TOP_LEVEL_AURI VARCHAR(80) CHARACTER SET utf8 NULL , ABC9 VARCHAR(255 ) CHARACTER SET utf8 NULL , INDEX(_URI ) USING HASH , INDEX(_LAST_UPDATE_DATE ) , INDEX(_PARENT_AURI ) USING HASH )
Aug 28, 2015 8:56:49 PM org.opendatakit.common.persistence.engine.mysql.DatastoreImpl assertRelation
INFO: create table success (before updateRelation): BUILD_35524174_CORE100
Aug 28, 2015 8:56:49 PM org.opendatakit.aggregate.parser.FormParserForJavaRosa orderlyDivideTable
INFO: Attempting to divide BUILD_35524174_CORE
Aug 28, 2015 8:56:49 PM org.opendatakit.common.persistence.engine.mysql.DatastoreImpl hasRelation
INFO: BUILD_35524174_CORE101 does not exist!
Aug 28, 2015 8:56:49 PM org.opendatakit.aggregate.parser.FormParserForJavaRosa orderlyDivideTable
INFO: Unable to cleave along groups; attempting phantom table! BUILD_35524174_CORE
Aug 28, 2015 8:56:49 PM org.opendatakit.aggregate.parser.FormParserForJavaRosa orderlyDivideTable
INFO: Created phantom for BUILD_35524174_CORE beginning at 608 with a total of 1 cleaved
Aug 28, 2015 8:56:49 PM org.opendatakit.aggregate.parser.FormParserForJavaRosa guardedInitHelper
SEVERE: Aborting form-creation due to fail-safe limit (100 attempts)!
Aug 28, 2015 8:56:49 PM org.opendatakit.aggregate.parser.FormParserForJavaRosa guardedInitHelper
WARNING: Aborting form-creation do to exception: org.opendatakit.aggregate.exception.ODKParseException: Unable to create form data tables after 100 attempts.. Datastore exceptions are expected in the following stack trace; other exceptions may indicate a problem:
org.opendatakit.aggregate.exception.ODKParseException: Unable to create form data tables after 100 attempts.
at org.opendatakit.aggregate.parser.FormParserForJavaRosa.guardedInitHelper(FormParserForJavaRosa.java:577)
at org.opendatakit.aggregate.parser.FormParserForJavaRosa.initHelper(FormParserForJavaRosa.java:236)
at org.opendatakit.aggregate.parser.FormParserForJavaRosa.(FormParserForJavaRosa.java:148)

(full catalina.out attached - warning, v large)

Mitch_S · August 28, 2015, 7:49pm

Yes, I would need to have the form. Send it to me directly (
mitchellsundt@gmail.com )

···

On Fri, Aug 28, 2015 at 12:29 PM, wrote:

Need to reopen this one, unfortunately.

We have been unable to upload the large form with our new server stack.
Details:
Aggregate 1.4.7
Tomcat 6
mysql Ver 14.14 Distrib 5.6.25
Java 8 (openjdk version "1.8.0_45-internal")
Ubuntu 15.04

The form definition that we're trying to upload is 1.4 MB (about 27400
lines). I can send you the form if that would be helpful.

Error as observed on Aggregate:
HTTP Status 400 - Problem parsing submission XML
org.opendatakit.aggregate.exception.ODKParseException: Error processing new
form: org.opendatakit.aggregate.exception.ODKParseException: Unable to
create form data tables after 100 attempts.

type Status report

message Problem parsing submission XML
org.opendatakit.aggregate.exception.ODKParseException: Error processing new
form: org.opendatakit.aggregate.exception.ODKParseException: Unable to
create form data tables after 100 attempts.

description The request sent by the client was syntactically incorrect.

Apache Tomcat/6.0.41

catalina.out logfile

INFO: Executing DROP TABLE fresh.BUILD_35524174_CORE by user
anonymousUser
Aug 28, 2015 8:56:48 PM
org.opendatakit.common.persistence.engine.mysql.DatastoreImpl dropRelation
WARNING: BUILD_35524174_CORE exception:
org.springframework.jdbc.BadSqlGrammarException: StatementCallback; bad SQL
grammar [DROP TABLE fresh.BUILD_35524174_CORE]; nested exception is
com.mysql.jdbc.exceptions.MySQLSyntaxErrorException: Unknown table
'fresh.BUILD_35524174_CORE'
Aug 28, 2015 8:56:48 PM
org.opendatakit.common.persistence.engine.mysql.DatastoreImpl assertRelation
INFO: before updateRelation: BUILD_35524174_CORE100
Aug 28, 2015 8:56:48 PM
org.opendatakit.common.persistence.engine.mysql.DatastoreImpl assertRelation
INFO: Attempting: CREATE TABLE fresh.BUILD_35524174_CORE100 ( _URI
VARCHAR(80) CHARACTER SET utf8 NOT NULL , _CREATOR_URI_USER VARCHAR(80)
CHARACTER SET utf8 NOT NULL , _CREATION_DATE DATETIME NOT NULL ,
_LAST_UPDATE_URI_USER VARCHAR(80) CHARACTER SET utf8 NULL ,
_LAST_UPDATE_DATE DATETIME NOT NULL , _PARENT_AURI VARCHAR(80)
CHARACTER SET utf8 NULL , _ORDINAL_NUMBER INTEGER(9 ) NOT NULL ,
_TOP_LEVEL_AURI VARCHAR(80) CHARACTER SET utf8 NULL , ABC9 VARCHAR(255
) CHARACTER SET utf8 NULL , INDEX(_URI ) USING HASH ,
INDEX(_LAST_UPDATE_DATE ) , INDEX(_PARENT_AURI ) USING HASH )
Aug 28, 2015 8:56:49 PM
org.opendatakit.common.persistence.engine.mysql.DatastoreImpl assertRelation
INFO: create table success (before updateRelation): BUILD_35524174_CORE100
Aug 28, 2015 8:56:49 PM
org.opendatakit.aggregate.parser.FormParserForJavaRosa orderlyDivideTable
INFO: Attempting to divide BUILD_35524174_CORE
Aug 28, 2015 8:56:49 PM
org.opendatakit.common.persistence.engine.mysql.DatastoreImpl hasRelation
INFO: BUILD_35524174_CORE101 does not exist!
Aug 28, 2015 8:56:49 PM
org.opendatakit.aggregate.parser.FormParserForJavaRosa orderlyDivideTable
INFO: Unable to cleave along groups; attempting phantom table!
BUILD_35524174_CORE
Aug 28, 2015 8:56:49 PM
org.opendatakit.aggregate.parser.FormParserForJavaRosa orderlyDivideTable
INFO: Created phantom for BUILD_35524174_CORE beginning at 608 with a
total of 1 cleaved
Aug 28, 2015 8:56:49 PM
org.opendatakit.aggregate.parser.FormParserForJavaRosa guardedInitHelper
SEVERE: Aborting form-creation due to fail-safe limit (100 attempts)!
Aug 28, 2015 8:56:49 PM
org.opendatakit.aggregate.parser.FormParserForJavaRosa guardedInitHelper
WARNING: Aborting form-creation do to exception:
org.opendatakit.aggregate.exception.ODKParseException: Unable to create
form data tables after 100 attempts.. Datastore exceptions are expected in
the following stack trace; other exceptions may indicate a problem:
org.opendatakit.aggregate.exception.ODKParseException: Unable to create
form data tables after 100 attempts.
at
org.opendatakit.aggregate.parser.FormParserForJavaRosa.guardedInitHelper(FormParserForJavaRosa.java:577)
at
org.opendatakit.aggregate.parser.FormParserForJavaRosa.initHelper(FormParserForJavaRosa.java:236)
at
org.opendatakit.aggregate.parser.FormParserForJavaRosa.(FormParserForJavaRosa.java:148)

(full catalina.out attached - warning, v large)

--

Post: opendatakit@googlegroups.com
Unsubscribe: opendatakit+unsubscribe@googlegroups.com
Options: http://groups.google.com/group/opendatakit?hl=en

You received this message because you are subscribed to the Google Groups
"ODK Community" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to opendatakit+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--
Mitch Sundt
Software Engineer
University of Washington
mitchellsundt@gmail.com

Colin · September 2, 2015, 8:59pm

Sent to your email directly, thank

···

On Friday, August 28, 2015 at 3:49:02 PM UTC-4, Mitch Sundt wrote: > Yes, I would need to have the form. Send it to me directly ( mitche...@gmail.com ) > > > > > On Fri, Aug 28, 2015 at 12:29 PM, wrote: > Need to reopen this one, unfortunately. > > > > We have been unable to upload the large form with our new server stack. Details: > > Aggregate 1.4.7 > > Tomcat 6 > > mysql Ver 14.14 Distrib 5.6.25 > > Java 8 (openjdk version "1.8.0_45-internal") > > Ubuntu 15.04 > > > > The form definition that we're trying to upload is 1.4 MB (about 27400 lines). I can send you the form if that would be helpful. > > > > > > Error as observed on Aggregate: > > HTTP Status 400 - Problem parsing submission XML org.opendatakit.aggregate.exception.ODKParseException: Error processing new form: org.opendatakit.aggregate.exception.ODKParseException: Unable to create form data tables after 100 attempts. > > > > type Status report > > > > message Problem parsing submission XML org.opendatakit.aggregate.exception.ODKParseException: Error processing new form: org.opendatakit.aggregate.exception.ODKParseException: Unable to create form data tables after 100 attempts. > > > > description The request sent by the client was syntactically incorrect. > > > > Apache Tomcat/6.0.41 > > > > > > > > catalina.out logfile > > > > INFO: Executing DROP TABLE `fresh`.`BUILD_35524174_CORE` by user anonymousUser > > Aug 28, 2015 8:56:48 PM org.opendatakit.common.persistence.engine.mysql.DatastoreImpl dropRelation > > WARNING: BUILD_35524174_CORE exception: org.springframework.jdbc.BadSqlGrammarException: StatementCallback; bad SQL grammar [DROP TABLE `fresh`.`BUILD_35524174_CORE`]; nested exception is com.mysql.jdbc.exceptions.MySQLSyntaxErrorException: Unknown table 'fresh.BUILD_35524174_CORE' > > Aug 28, 2015 8:56:48 PM org.opendatakit.common.persistence.engine.mysql.DatastoreImpl assertRelation > > INFO: before updateRelation: BUILD_35524174_CORE100 > > Aug 28, 2015 8:56:48 PM org.opendatakit.common.persistence.engine.mysql.DatastoreImpl assertRelation > > INFO: Attempting: CREATE TABLE `fresh`.`BUILD_35524174_CORE100` ( `_URI` VARCHAR(80) CHARACTER SET utf8 NOT NULL , `_CREATOR_URI_USER` VARCHAR(80) CHARACTER SET utf8 NOT NULL , `_CREATION_DATE` DATETIME NOT NULL , `_LAST_UPDATE_URI_USER` VARCHAR(80) CHARACTER SET utf8 NULL , `_LAST_UPDATE_DATE` DATETIME NOT NULL , `_PARENT_AURI` VARCHAR(80) CHARACTER SET utf8 NULL , `_ORDINAL_NUMBER` INTEGER(9 ) NOT NULL , `_TOP_LEVEL_AURI` VARCHAR(80) CHARACTER SET utf8 NULL , `ABC9` VARCHAR(255 ) CHARACTER SET utf8 NULL , INDEX(`_URI` ) USING HASH , INDEX(`_LAST_UPDATE_DATE` ) , INDEX(`_PARENT_AURI` ) USING HASH ) > > Aug 28, 2015 8:56:49 PM org.opendatakit.common.persistence.engine.mysql.DatastoreImpl assertRelation > > INFO: create table success (before updateRelation): BUILD_35524174_CORE100 > > Aug 28, 2015 8:56:49 PM org.opendatakit.aggregate.parser.FormParserForJavaRosa orderlyDivideTable > > INFO: Attempting to divide BUILD_35524174_CORE > > Aug 28, 2015 8:56:49 PM org.opendatakit.common.persistence.engine.mysql.DatastoreImpl hasRelation > > INFO: BUILD_35524174_CORE101 does not exist! > > Aug 28, 2015 8:56:49 PM org.opendatakit.aggregate.parser.FormParserForJavaRosa orderlyDivideTable > > INFO: Unable to cleave along groups; attempting phantom table! BUILD_35524174_CORE > > Aug 28, 2015 8:56:49 PM org.opendatakit.aggregate.parser.FormParserForJavaRosa orderlyDivideTable > > INFO: Created phantom for BUILD_35524174_CORE beginning at 608 with a total of 1 cleaved > > Aug 28, 2015 8:56:49 PM org.opendatakit.aggregate.parser.FormParserForJavaRosa guardedInitHelper > > SEVERE: Aborting form-creation due to fail-safe limit (100 attempts)! > > Aug 28, 2015 8:56:49 PM org.opendatakit.aggregate.parser.FormParserForJavaRosa guardedInitHelper > > WARNING: Aborting form-creation do to exception: org.opendatakit.aggregate.exception.ODKParseException: Unable to create form data tables after 100 attempts.. Datastore exceptions are expected in the following stack trace; other exceptions may indicate a problem: > > org.opendatakit.aggregate.exception.ODKParseException: Unable to create form data tables after 100 attempts. > > at org.opendatakit.aggregate.parser.FormParserForJavaRosa.guardedInitHelper(FormParserForJavaRosa.java:577) > > at org.opendatakit.aggregate.parser.FormParserForJavaRosa.initHelper(FormParserForJavaRosa.java:236) > > at org.opendatakit.aggregate.parser.FormParserForJavaRosa.(FormParserForJavaRosa.java:148) > > > > (full catalina.out attached - warning, v large) > > > > > > -- > > -- > > Post: opend...@googlegroups.com > > Unsubscribe: opendatakit...@googlegroups.com > > Options: http://groups.google.com/group/opendatakit?hl=en > > > > --- > > You received this message because you are subscribed to the Google Groups "ODK Community" group. > > To unsubscribe from this group and stop receiving emails from it, send an email to opendatakit...@googlegroups.com. > > For more options, visit https://groups.google.com/d/optout. > > > > > > -- > > Mitch Sundt > Software Engineer > University of Washington > mitche...@gmail.com

Colin · September 8, 2015, 6:09pm

Just wanted to bump this in case it got lost in the shuffle...
(forms have been emailed directly to mitchellsundt at gmail)

···

On Friday, August 28, 2015 at 3:49:02 PM UTC-4, Mitch Sundt wrote: > Yes, I would need to have the form. Send it to me directly ( mitche...@gmail.com ) > > > > > On Fri, Aug 28, 2015 at 12:29 PM, wrote: > Need to reopen this one, unfortunately. > > > > We have been unable to upload the large form with our new server stack. Details: > > Aggregate 1.4.7 > > Tomcat 6 > > mysql Ver 14.14 Distrib 5.6.25 > > Java 8 (openjdk version "1.8.0_45-internal") > > Ubuntu 15.04 > > > > The form definition that we're trying to upload is 1.4 MB (about 27400 lines). I can send you the form if that would be helpful. > > > > > > Error as observed on Aggregate: > > HTTP Status 400 - Problem parsing submission XML org.opendatakit.aggregate.exception.ODKParseException: Error processing new form: org.opendatakit.aggregate.exception.ODKParseException: Unable to create form data tables after 100 attempts. > > > > type Status report > > > > message Problem parsing submission XML org.opendatakit.aggregate.exception.ODKParseException: Error processing new form: org.opendatakit.aggregate.exception.ODKParseException: Unable to create form data tables after 100 attempts. > > > > description The request sent by the client was syntactically incorrect. > > > > Apache Tomcat/6.0.41 > > > > > > > > catalina.out logfile > > > > INFO: Executing DROP TABLE `fresh`.`BUILD_35524174_CORE` by user anonymousUser > > Aug 28, 2015 8:56:48 PM org.opendatakit.common.persistence.engine.mysql.DatastoreImpl dropRelation > > WARNING: BUILD_35524174_CORE exception: org.springframework.jdbc.BadSqlGrammarException: StatementCallback; bad SQL grammar [DROP TABLE `fresh`.`BUILD_35524174_CORE`]; nested exception is com.mysql.jdbc.exceptions.MySQLSyntaxErrorException: Unknown table 'fresh.BUILD_35524174_CORE' > > Aug 28, 2015 8:56:48 PM org.opendatakit.common.persistence.engine.mysql.DatastoreImpl assertRelation > > INFO: before updateRelation: BUILD_35524174_CORE100 > > Aug 28, 2015 8:56:48 PM org.opendatakit.common.persistence.engine.mysql.DatastoreImpl assertRelation > > INFO: Attempting: CREATE TABLE `fresh`.`BUILD_35524174_CORE100` ( `_URI` VARCHAR(80) CHARACTER SET utf8 NOT NULL , `_CREATOR_URI_USER` VARCHAR(80) CHARACTER SET utf8 NOT NULL , `_CREATION_DATE` DATETIME NOT NULL , `_LAST_UPDATE_URI_USER` VARCHAR(80) CHARACTER SET utf8 NULL , `_LAST_UPDATE_DATE` DATETIME NOT NULL , `_PARENT_AURI` VARCHAR(80) CHARACTER SET utf8 NULL , `_ORDINAL_NUMBER` INTEGER(9 ) NOT NULL , `_TOP_LEVEL_AURI` VARCHAR(80) CHARACTER SET utf8 NULL , `ABC9` VARCHAR(255 ) CHARACTER SET utf8 NULL , INDEX(`_URI` ) USING HASH , INDEX(`_LAST_UPDATE_DATE` ) , INDEX(`_PARENT_AURI` ) USING HASH ) > > Aug 28, 2015 8:56:49 PM org.opendatakit.common.persistence.engine.mysql.DatastoreImpl assertRelation > > INFO: create table success (before updateRelation): BUILD_35524174_CORE100 > > Aug 28, 2015 8:56:49 PM org.opendatakit.aggregate.parser.FormParserForJavaRosa orderlyDivideTable > > INFO: Attempting to divide BUILD_35524174_CORE > > Aug 28, 2015 8:56:49 PM org.opendatakit.common.persistence.engine.mysql.DatastoreImpl hasRelation > > INFO: BUILD_35524174_CORE101 does not exist! > > Aug 28, 2015 8:56:49 PM org.opendatakit.aggregate.parser.FormParserForJavaRosa orderlyDivideTable > > INFO: Unable to cleave along groups; attempting phantom table! BUILD_35524174_CORE > > Aug 28, 2015 8:56:49 PM org.opendatakit.aggregate.parser.FormParserForJavaRosa orderlyDivideTable > > INFO: Created phantom for BUILD_35524174_CORE beginning at 608 with a total of 1 cleaved > > Aug 28, 2015 8:56:49 PM org.opendatakit.aggregate.parser.FormParserForJavaRosa guardedInitHelper > > SEVERE: Aborting form-creation due to fail-safe limit (100 attempts)! > > Aug 28, 2015 8:56:49 PM org.opendatakit.aggregate.parser.FormParserForJavaRosa guardedInitHelper > > WARNING: Aborting form-creation do to exception: org.opendatakit.aggregate.exception.ODKParseException: Unable to create form data tables after 100 attempts.. Datastore exceptions are expected in the following stack trace; other exceptions may indicate a problem: > > org.opendatakit.aggregate.exception.ODKParseException: Unable to create form data tables after 100 attempts. > > at org.opendatakit.aggregate.parser.FormParserForJavaRosa.guardedInitHelper(FormParserForJavaRosa.java:577) > > at org.opendatakit.aggregate.parser.FormParserForJavaRosa.initHelper(FormParserForJavaRosa.java:236) > > at org.opendatakit.aggregate.parser.FormParserForJavaRosa.(FormParserForJavaRosa.java:148) > > > > (full catalina.out attached - warning, v large) > > > > > > -- > > -- > > Post: opend...@googlegroups.com > > Unsubscribe: opendatakit...@googlegroups.com > > Options: http://groups.google.com/group/opendatakit?hl=en > > > > --- > > You received this message because you are subscribed to the Google Groups "ODK Community" group. > > To unsubscribe from this group and stop receiving emails from it, send an email to opendatakit...@googlegroups.com. > > For more options, visit https://groups.google.com/d/optout. > > > > > > -- > > Mitch Sundt > Software Engineer > University of Washington > mitche...@gmail.com

Mitch_S · September 8, 2015, 6:32pm

I haven't had time to fully explore what is going on.

My initial thought is that you can probably work around this by placing

begin group
end group

around the various groups of questions. I.e., all the AZT1... AZT12...
questions could be surrounded by a 'AZT' group.
etc.

And perhaps even placing meaningless begin group... end group around sets
of these groups.

Adding these groups (these are not field-list groups -- just simple groups
without any relevance conditions or other features) should not alter the
logic of your form. It will push the fields down a layer in the XML
structure and add a crumb to the crumb trail at the top of your ODK Collect
screen.

These groups will aid the table-splitting algorithm by creating bigger
chunks of things to shuffle around.

The server has 100 tries to successfully split the fields of a survey into
the number of tables required to contain its values. Because of its size,
your form needs at least 5 tables.

The splitting is done naively by trying to fit one or more groups of
questions into a single table. If the group(s) don't fit, it splits the set
of groups in half, and, if it is a single group, it splits the number of
questions across two different tables. If the splitting does not work, it
then further subdivides the problem until it can create the table.

Because it retries after the first failure, this process will, I think,
fail if the survey requires more than 8 tables (2^8 = 128 which is greater
than 100) unless it is guided to be more efficient in how it determines the
split by the use of groups.

···

On Tue, Sep 8, 2015 at 11:09 AM, wrote:

Just wanted to bump this in case it got lost in the shuffle...
(forms have been emailed directly to mitchellsundt at gmail)

On Friday, August 28, 2015 at 3:49:02 PM UTC-4, Mitch Sundt wrote:
Yes, I would need to have the form. Send it to me directly (
mitche...@gmail.com )

On Fri, Aug 28, 2015 at 12:29 PM, colind...@gmail.com wrote:
Need to reopen this one, unfortunately.

We have been unable to upload the large form with our new server stack.
Details:

Aggregate 1.4.7

Tomcat 6

mysql Ver 14.14 Distrib 5.6.25

Java 8 (openjdk version "1.8.0_45-internal")

Ubuntu 15.04

The form definition that we're trying to upload is 1.4 MB (about 27400
lines). I can send you the form if that would be helpful.

Error as observed on Aggregate:

HTTP Status 400 - Problem parsing submission XML
org.opendatakit.aggregate.exception.ODKParseException: Error processing new
form: org.opendatakit.aggregate.exception.ODKParseException: Unable to
create form data tables after 100 attempts.

type Status report

message Problem parsing submission XML
org.opendatakit.aggregate.exception.ODKParseException: Error processing new
form: org.opendatakit.aggregate.exception.ODKParseException: Unable to
create form data tables after 100 attempts.

description The request sent by the client was syntactically incorrect.

Apache Tomcat/6.0.41

catalina.out logfile

INFO: Executing DROP TABLE fresh.BUILD_35524174_CORE by user
anonymousUser

Aug 28, 2015 8:56:48 PM
org.opendatakit.common.persistence.engine.mysql.DatastoreImpl dropRelation

WARNING: BUILD_35524174_CORE exception:
org.springframework.jdbc.BadSqlGrammarException: StatementCallback; bad SQL
grammar [DROP TABLE fresh.BUILD_35524174_CORE]; nested exception is
com.mysql.jdbc.exceptions.MySQLSyntaxErrorException: Unknown table
'fresh.BUILD_35524174_CORE'

Aug 28, 2015 8:56:48 PM
org.opendatakit.common.persistence.engine.mysql.DatastoreImpl assertRelation

INFO: before updateRelation: BUILD_35524174_CORE100

Aug 28, 2015 8:56:48 PM
org.opendatakit.common.persistence.engine.mysql.DatastoreImpl assertRelation

INFO: Attempting: CREATE TABLE fresh.BUILD_35524174_CORE100 ( _URI
VARCHAR(80) CHARACTER SET utf8 NOT NULL , _CREATOR_URI_USER VARCHAR(80)
CHARACTER SET utf8 NOT NULL , _CREATION_DATE DATETIME NOT NULL ,
_LAST_UPDATE_URI_USER VARCHAR(80) CHARACTER SET utf8 NULL ,
_LAST_UPDATE_DATE DATETIME NOT NULL , _PARENT_AURI VARCHAR(80)
CHARACTER SET utf8 NULL , _ORDINAL_NUMBER INTEGER(9 ) NOT NULL ,
_TOP_LEVEL_AURI VARCHAR(80) CHARACTER SET utf8 NULL , ABC9 VARCHAR(255
) CHARACTER SET utf8 NULL , INDEX(_URI ) USING HASH ,
INDEX(_LAST_UPDATE_DATE ) , INDEX(_PARENT_AURI ) USING HASH )

Aug 28, 2015 8:56:49 PM
org.opendatakit.common.persistence.engine.mysql.DatastoreImpl assertRelation

INFO: create table success (before updateRelation):
BUILD_35524174_CORE100

Aug 28, 2015 8:56:49 PM
org.opendatakit.aggregate.parser.FormParserForJavaRosa orderlyDivideTable

INFO: Attempting to divide BUILD_35524174_CORE

Aug 28, 2015 8:56:49 PM
org.opendatakit.common.persistence.engine.mysql.DatastoreImpl hasRelation

INFO: BUILD_35524174_CORE101 does not exist!

Aug 28, 2015 8:56:49 PM
org.opendatakit.aggregate.parser.FormParserForJavaRosa orderlyDivideTable

INFO: Unable to cleave along groups; attempting phantom table!
BUILD_35524174_CORE

Aug 28, 2015 8:56:49 PM
org.opendatakit.aggregate.parser.FormParserForJavaRosa orderlyDivideTable

INFO: Created phantom for BUILD_35524174_CORE beginning at 608 with a
total of 1 cleaved

Aug 28, 2015 8:56:49 PM
org.opendatakit.aggregate.parser.FormParserForJavaRosa guardedInitHelper

SEVERE: Aborting form-creation due to fail-safe limit (100 attempts)!

Aug 28, 2015 8:56:49 PM
org.opendatakit.aggregate.parser.FormParserForJavaRosa guardedInitHelper

WARNING: Aborting form-creation do to exception:
org.opendatakit.aggregate.exception.ODKParseException: Unable to create
form data tables after 100 attempts.. Datastore exceptions are expected in
the following stack trace; other exceptions may indicate a problem:

org.opendatakit.aggregate.exception.ODKParseException: Unable to create
form data tables after 100 attempts.
    at
org.opendatakit.aggregate.parser.FormParserForJavaRosa.guardedInitHelper(FormParserForJavaRosa.java:577)
    at
org.opendatakit.aggregate.parser.FormParserForJavaRosa.initHelper(FormParserForJavaRosa.java:236)
    at
org.opendatakit.aggregate.parser.FormParserForJavaRosa.(FormParserForJavaRosa.java:148)

(full catalina.out attached - warning, v large)

--

--

Post: opend...@googlegroups.com

Unsubscribe: opendatakit...@googlegroups.com

Options: http://groups.google.com/group/opendatakit?hl=en

You received this message because you are subscribed to the Google
Groups "ODK Community" group.

To unsubscribe from this group and stop receiving emails from it, send
an email to opendatakit...@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--

Mitch Sundt
Software Engineer
University of Washington
mitche...@gmail.com

--

Post: opendatakit@googlegroups.com
Unsubscribe: opendatakit+unsubscribe@googlegroups.com
Options: http://groups.google.com/group/opendatakit?hl=en

You received this message because you are subscribed to the Google Groups
"ODK Community" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to opendatakit+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--
Mitch Sundt
Software Engineer
University of Washington
mitchellsundt@gmail.com

Colin · September 8, 2015, 9:53pm

This is really interesting to know, thanks.

And in even better news, your suggestion worked! I wrapped all groups (eg AZT1_group ... AZT12_group) in 'super'-groups (this took some regex-fu, which is very much not my specialty), and the server seems to be happy to eat it!

This is a huge relief! Our data collection can now continue.

Thanks so much for your continued help on this. We very much appreciate it!

···

On Tuesday, September 8, 2015 at 2:32:54 PM UTC-4, Mitch Sundt wrote: > I haven't had time to fully explore what is going on. > > > My initial thought is that you can probably work around this by placing > > > begin group > end group > > > around the various groups of questions. I.e., all the AZT1... AZT12... questions could be surrounded by a 'AZT' group. > etc. > > > > And perhaps even placing meaningless begin group... end group around sets of these groups. > > > Adding these groups (these are not field-list groups -- just simple groups without any relevance conditions or other features) should not alter the logic of your form. It will push the fields down a layer in the XML structure and add a crumb to the crumb trail at the top of your ODK Collect screen. > > > These groups will aid the table-splitting algorithm by creating bigger chunks of things to shuffle around. > > > The server has 100 tries to successfully split the fields of a survey into the number of tables required to contain its values. Because of its size, your form needs at least 5 tables. > > > The splitting is done naively by trying to fit one or more groups of questions into a single table. If the group(s) don't fit, it splits the set of groups in half, and, if it is a single group, it splits the number of questions across two different tables. If the splitting does not work, it then further subdivides the problem until it can create the table. > > > Because it retries after the first failure, this process will, I think, fail if the survey requires more than 8 tables (2^8 = 128 which is greater than 100) unless it is guided to be more efficient in how it determines the split by the use of groups. > > > > > On Tue, Sep 8, 2015 at 11:09 AM, wrote: > Just wanted to bump this in case it got lost in the shuffle... > > (forms have been emailed directly to mitchellsundt at gmail) > > > > > > On Friday, August 28, 2015 at 3:49:02 PM UTC-4, Mitch Sundt wrote: > > > Yes, I would need to have the form. Send it to me directly ( mitche...@gmail.com ) > > > > > > > > > > > > > > > > > On Fri, Aug 28, 2015 at 12:29 PM, wrote: > > > Need to reopen this one, unfortunately. > > > > > > > > > > > > We have been unable to upload the large form with our new server stack. Details: > > > > > > Aggregate 1.4.7 > > > > > > Tomcat 6 > > > > > > mysql Ver 14.14 Distrib 5.6.25 > > > > > > Java 8 (openjdk version "1.8.0_45-internal") > > > > > > Ubuntu 15.04 > > > > > > > > > > > > The form definition that we're trying to upload is 1.4 MB (about 27400 lines). I can send you the form if that would be helpful. > > > > > > > > > > > > > > > > > > Error as observed on Aggregate: > > > > > > HTTP Status 400 - Problem parsing submission XML org.opendatakit.aggregate.exception.ODKParseException: Error processing new form: org.opendatakit.aggregate.exception.ODKParseException: Unable to create form data tables after 100 attempts. > > > > > > > > > > > > type Status report > > > > > > > > > > > > message Problem parsing submission XML org.opendatakit.aggregate.exception.ODKParseException: Error processing new form: org.opendatakit.aggregate.exception.ODKParseException: Unable to create form data tables after 100 attempts. > > > > > > > > > > > > description The request sent by the client was syntactically incorrect. > > > > > > > > > > > > Apache Tomcat/6.0.41 > > > > > > > > > > > > > > > > > > > > > > > > catalina.out logfile > > > > > > > > > > > > INFO: Executing DROP TABLE `fresh`.`BUILD_35524174_CORE` by user anonymousUser > > > > > > Aug 28, 2015 8:56:48 PM org.opendatakit.common.persistence.engine.mysql.DatastoreImpl dropRelation > > > > > > WARNING: BUILD_35524174_CORE exception: org.springframework.jdbc.BadSqlGrammarException: StatementCallback; bad SQL grammar [DROP TABLE `fresh`.`BUILD_35524174_CORE`]; nested exception is com.mysql.jdbc.exceptions.MySQLSyntaxErrorException: Unknown table 'fresh.BUILD_35524174_CORE' > > > > > > Aug 28, 2015 8:56:48 PM org.opendatakit.common.persistence.engine.mysql.DatastoreImpl assertRelation > > > > > > INFO: before updateRelation: BUILD_35524174_CORE100 > > > > > > Aug 28, 2015 8:56:48 PM org.opendatakit.common.persistence.engine.mysql.DatastoreImpl assertRelation > > > > > > INFO: Attempting: CREATE TABLE `fresh`.`BUILD_35524174_CORE100` ( `_URI` VARCHAR(80) CHARACTER SET utf8 NOT NULL , `_CREATOR_URI_USER` VARCHAR(80) CHARACTER SET utf8 NOT NULL , `_CREATION_DATE` DATETIME NOT NULL , `_LAST_UPDATE_URI_USER` VARCHAR(80) CHARACTER SET utf8 NULL , `_LAST_UPDATE_DATE` DATETIME NOT NULL , `_PARENT_AURI` VARCHAR(80) CHARACTER SET utf8 NULL , `_ORDINAL_NUMBER` INTEGER(9 ) NOT NULL , `_TOP_LEVEL_AURI` VARCHAR(80) CHARACTER SET utf8 NULL , `ABC9` VARCHAR(255 ) CHARACTER SET utf8 NULL , INDEX(`_URI` ) USING HASH , INDEX(`_LAST_UPDATE_DATE` ) , INDEX(`_PARENT_AURI` ) USING HASH ) > > > > > > Aug 28, 2015 8:56:49 PM org.opendatakit.common.persistence.engine.mysql.DatastoreImpl assertRelation > > > > > > INFO: create table success (before updateRelation): BUILD_35524174_CORE100 > > > > > > Aug 28, 2015 8:56:49 PM org.opendatakit.aggregate.parser.FormParserForJavaRosa orderlyDivideTable > > > > > > INFO: Attempting to divide BUILD_35524174_CORE > > > > > > Aug 28, 2015 8:56:49 PM org.opendatakit.common.persistence.engine.mysql.DatastoreImpl hasRelation > > > > > > INFO: BUILD_35524174_CORE101 does not exist! > > > > > > Aug 28, 2015 8:56:49 PM org.opendatakit.aggregate.parser.FormParserForJavaRosa orderlyDivideTable > > > > > > INFO: Unable to cleave along groups; attempting phantom table! BUILD_35524174_CORE > > > > > > Aug 28, 2015 8:56:49 PM org.opendatakit.aggregate.parser.FormParserForJavaRosa orderlyDivideTable > > > > > > INFO: Created phantom for BUILD_35524174_CORE beginning at 608 with a total of 1 cleaved > > > > > > Aug 28, 2015 8:56:49 PM org.opendatakit.aggregate.parser.FormParserForJavaRosa guardedInitHelper > > > > > > SEVERE: Aborting form-creation due to fail-safe limit (100 attempts)! > > > > > > Aug 28, 2015 8:56:49 PM org.opendatakit.aggregate.parser.FormParserForJavaRosa guardedInitHelper > > > > > > WARNING: Aborting form-creation do to exception: org.opendatakit.aggregate.exception.ODKParseException: Unable to create form data tables after 100 attempts.. Datastore exceptions are expected in the following stack trace; other exceptions may indicate a problem: > > > > > > org.opendatakit.aggregate.exception.ODKParseException: Unable to create form data tables after 100 attempts. > > > > > > at org.opendatakit.aggregate.parser.FormParserForJavaRosa.guardedInitHelper(FormParserForJavaRosa.java:577) > > > > > > at org.opendatakit.aggregate.parser.FormParserForJavaRosa.initHelper(FormParserForJavaRosa.java:236) > > > > > > at org.opendatakit.aggregate.parser.FormParserForJavaRosa.(FormParserForJavaRosa.java:148) > > > > > > > > > > > > (full catalina.out attached - warning, v large) > > > > > > > > > > > > > > > > > > -- > > > > > > -- > > > > > > Post: opend...@googlegroups.com > > > > > > Unsubscribe: opendatakit...@googlegroups.com > > > > > > Options: http://groups.google.com/group/opendatakit?hl=en > > > > > > > > > > > > --- > > > > > > You received this message because you are subscribed to the Google Groups "ODK Community" group. > > > > > > To unsubscribe from this group and stop receiving emails from it, send an email to opendatakit...@googlegroups.com. > > > > > > For more options, visit https://groups.google.com/d/optout. > > > > > > > > > > > > > > > > > > -- > > > > > > Mitch Sundt > > > Software Engineer > > > University of Washington > > > mitche...@gmail.com > > > > > > -- > > -- > > Post: opend...@googlegroups.com > > Unsubscribe: opendatakit...@googlegroups.com > > Options: http://groups.google.com/group/opendatakit?hl=en > > > > --- > > You received this message because you are subscribed to the Google Groups "ODK Community" group. > > To unsubscribe from this group and stop receiving emails from it, send an email to opendatakit...@googlegroups.com. > > For more options, visit https://groups.google.com/d/optout. > > > > > > -- > > Mitch Sundt > Software Engineer > University of Washington > mitche...@gmail.com