ODK Aggregate : Failure - will retry later while trying to export CSV file

Pablo_Rodriguez · May 3, 2016, 12:10pm

Hi guys,

I´m kind of new here, but I already have this problem:

On ODK Aggregate, when I want to export my data to .csv file, It´s showing me an error (Failure - will retry later). it was working fine but I guess the DDBB reach a certain volume that made it fail.

I have read that it is related to global max_allowed_packet which has to be set to a higher value, with the following expression

set global max_allowed_packet = 1073741824;

I have my ODK aggregate deployed in a google app engine, so i tied to type that in the cloud shell but nothing happened, I still have the same problem, and as you can imaging it´s the worst moment to happend...

I have a excel macro that reads the csv file and download the media files, renames and save everything in a file, so then i can open each picture from a link in an excel spreadsheet.

Now I can only download the the CSV using ODK briefcase by:
1- Pulling all the data.
2- Exporting the CSV (I can only do it if I firstly pull the data)

The problem is that by doing this I have to download the media files twice (in briefcase and in my macro), and the CSV file exported by briefcase contains the local link in my computer to the media files, but not the cloud link to the media files, which in my case it is super convenient. I figure out that the link to the media is a concatenation of the ID number and some other things, so I manage to bypass this problem, but I had to change my whole procedure in the middle of the work, making it much more manual and not allowing me to use the tools I developped to automate all this work.

Anyone knows how can I in ODK aggregate deployed in google app engine execute this procedure?
set global max_allowed_packet = 1073741824;

IF that really is the problem, which i hope it is.

Thank you very much in advance.

Regards,

Pablo

yanokwa · May 3, 2016, 12:56pm

Pablo,

max_allowed_packet is a MySQL command and so only works on self-hosted
solutions backed my MySQL. Since you are on Google App Engine, that
command will not work.

One easy to thing to try is to stream your data to Google FusionTables
or Sheets and then download the CSV those systems generate.

Yaw

···

-- Need ODK consultants? Nafundi provides form design, server setup, in-field training, and software development for ODK. Go to https://nafundi.com to get started.

On Tue, May 3, 2016 at 2:10 PM, nuncaestardesillegaspronto@gmail.com wrote:

Hi guys,

I´m kind of new here, but I already have this problem:

On ODK Aggregate, when I want to export my data to .csv file, It´s showing me an error (Failure - will retry later). it was working fine but I guess the DDBB reach a certain volume that made it fail.

I have read that it is related to global max_allowed_packet which has to be set to a higher value, with the following expression

set global max_allowed_packet = 1073741824;

I have my ODK aggregate deployed in a google app engine, so i tied to type that in the cloud shell but nothing happened, I still have the same problem, and as you can imaging it´s the worst moment to happend...

I have a excel macro that reads the csv file and download the media files, renames and save everything in a file, so then i can open each picture from a link in an excel spreadsheet.

Now I can only download the the CSV using ODK briefcase by:
1- Pulling all the data.
2- Exporting the CSV (I can only do it if I firstly pull the data)

The problem is that by doing this I have to download the media files twice (in briefcase and in my macro), and the CSV file exported by briefcase contains the local link in my computer to the media files, but not the cloud link to the media files, which in my case it is super convenient. I figure out that the link to the media is a concatenation of the ID number and some other things, so I manage to bypass this problem, but I had to change my whole procedure in the middle of the work, making it much more manual and not allowing me to use the tools I developped to automate all this work.

Anyone knows how can I in ODK aggregate deployed in google app engine execute this procedure?
set global max_allowed_packet = 1073741824;

IF that really is the problem, which i hope it is.

Thank you very much in advance.

Regards,

Pablo

--

Post: opendatakit@googlegroups.com
Unsubscribe: opendatakit+unsubscribe@googlegroups.com
Options: http://groups.google.com/group/opendatakit?hl=en

You received this message because you are subscribed to the Google Groups "ODK Community" group.
To unsubscribe from this group and stop receiving emails from it, send an email to opendatakit+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Pablo_Rodriguez · May 6, 2016, 10:30am

One more thing, everytime I pull information with briefcase it downloads the whole project instead of just downloading the new entries,that could be something easy to improve in future versions, my excel macro that downloads everything from the csv does that it is muc more convenient this way.

Regards,

Pablo

···

El martes, 3 de mayo de 2016, 14:10:28 (UTC+2), nuncaestardes...@gmail.com escribió: > Hi guys, > > I´m kind of new here, but I already have this problem: > > On ODK Aggregate, when I want to export my data to .csv file, It´s showing me an error (Failure - will retry later). it was working fine but I guess the DDBB reach a certain volume that made it fail. > > I have read that it is related to global max_allowed_packet which has to be set to a higher value, with the following expression > > set global max_allowed_packet = 1073741824; > > I have my ODK aggregate deployed in a google app engine, so i tied to type that in the cloud shell but nothing happened, I still have the same problem, and as you can imaging it´s the worst moment to happend... > > I have a excel macro that reads the csv file and download the media files, renames and save everything in a file, so then i can open each picture from a link in an excel spreadsheet. > > Now I can only download the the CSV using ODK briefcase by: > 1- Pulling all the data. > 2- Exporting the CSV (I can only do it if I firstly pull the data) > > The problem is that by doing this I have to download the media files twice (in briefcase and in my macro), and the CSV file exported by briefcase contains the local link in my computer to the media files, but not the cloud link to the media files, which in my case it is super convenient. I figure out that the link to the media is a concatenation of the ID number and some other things, so I manage to bypass this problem, but I had to change my whole procedure in the middle of the work, making it much more manual and not allowing me to use the tools I developped to automate all this work. > > Anyone knows how can I in ODK aggregate deployed in google app engine execute this procedure? > set global max_allowed_packet = 1073741824; > > IF that really is the problem, which i hope it is. > > Thank you very much in advance. > > Regards, > > Pablo

Pablo_Rodriguez · May 3, 2016, 2:53pm

Thank you Yaw, I´m publising to google spreadsheets, because I am not familiar with fusion tables, it´s going slow, but it´s going.

Thanks,

Pablo

···

El martes, 3 de mayo de 2016, 14:57:20 (UTC+2), Yaw Anokwa escribió: > Pablo, > > max_allowed_packet is a MySQL command and so only works on self-hosted > solutions backed my MySQL. Since you are on Google App Engine, that > command will not work. > > One easy to thing to try is to stream your data to Google FusionTables > or Sheets and then download the CSV those systems generate. > > Yaw > -- > Need ODK consultants? Nafundi provides form design, server setup, > in-field training, and software development for ODK. Go to > https://nafundi.com to get started. > > On Tue, May 3, 2016 at 2:10 PM, wrote: > > Hi guys, > > > > I´m kind of new here, but I already have this problem: > > > > On ODK Aggregate, when I want to export my data to .csv file, It´s showing me an error (Failure - will retry later). it was working fine but I guess the DDBB reach a certain volume that made it fail. > > > > I have read that it is related to global max_allowed_packet which has to be set to a higher value, with the following expression > > > > set global max_allowed_packet = 1073741824; > > > > I have my ODK aggregate deployed in a google app engine, so i tied to type that in the cloud shell but nothing happened, I still have the same problem, and as you can imaging it´s the worst moment to happend... > > > > I have a excel macro that reads the csv file and download the media files, renames and save everything in a file, so then i can open each picture from a link in an excel spreadsheet. > > > > Now I can only download the the CSV using ODK briefcase by: > > 1- Pulling all the data. > > 2- Exporting the CSV (I can only do it if I firstly pull the data) > > > > The problem is that by doing this I have to download the media files twice (in briefcase and in my macro), and the CSV file exported by briefcase contains the local link in my computer to the media files, but not the cloud link to the media files, which in my case it is super convenient. I figure out that the link to the media is a concatenation of the ID number and some other things, so I manage to bypass this problem, but I had to change my whole procedure in the middle of the work, making it much more manual and not allowing me to use the tools I developped to automate all this work. > > > > Anyone knows how can I in ODK aggregate deployed in google app engine execute this procedure? > > set global max_allowed_packet = 1073741824; > > > > IF that really is the problem, which i hope it is. > > > > Thank you very much in advance. > > > > Regards, > > > > Pablo > > > > -- > > -- > > Post: opendatakit@googlegroups.com > > Unsubscribe: opendatakit+unsubscribe@googlegroups.com > > Options: http://groups.google.com/group/opendatakit?hl=en > > > > --- > > You received this message because you are subscribed to the Google Groups "ODK Community" group. > > To unsubscribe from this group and stop receiving emails from it, send an email to opendatakit+unsubscribe@googlegroups.com. > > For more options, visit https://groups.google.com/d/optout.

Mitch_S · May 3, 2016, 6:18pm

It is more likely that you are running out of memory on the ODK Aggregate
server.

When you Export to CSV, the entire dataset must be held in memory while the
CSV is generated. At some point that fails, and you need to move onto
either ODK Briefcase and its export functionality or to a publisher.

You can change the size of your servers by editing the files within the
ODKAggregate directories (this assumes you have upgraded to ODK Aggregate
1.4.8 -- you cannot do this on earlier versions).

To change the size of the server that affects the Export to CSV feature,
you must edit:

ODKAggregate\background\WEB-INF\appengine-web.xml

(using Notepad -- not Word)

Then change the B2 line.

You want to change B2 to a larger instance class.

Google (apparently) no longer wants to provide documentation on what their
class names mean.

The available classes are:

B1, B2, B4, B4_1G, and B8

If memory serves, these roughly correspond to 128MB, 256MB, 512MB, 1GB and
1GB of available memory.

So if you change from B2 to B4 (and re-upload your ODK Aggregate
configuration), you will double your JVM size, which will allow you to emit
2x more data via CSV.

···

On Tue, May 3, 2016 at 7:53 AM, wrote:

Thank you Yaw, I´m publising to google spreadsheets, because I am not
familiar with fusion tables, it´s going slow, but it´s going.

Thanks,

Pablo

El martes, 3 de mayo de 2016, 14:57:20 (UTC+2), Yaw Anokwa escribió:

Pablo,

max_allowed_packet is a MySQL command and so only works on self-hosted
solutions backed my MySQL. Since you are on Google App Engine, that
command will not work.

One easy to thing to try is to stream your data to Google FusionTables
or Sheets and then download the CSV those systems generate.

Yaw

Need ODK consultants? Nafundi provides form design, server setup,
in-field training, and software development for ODK. Go to
https://nafundi.com to get started.

On Tue, May 3, 2016 at 2:10 PM, nuncaestardesillegaspronto@gmail.com wrote:

Hi guys,

I´m kind of new here, but I already have this problem:

On ODK Aggregate, when I want to export my data to .csv file, It´s
showing me an error (Failure - will retry later). it was working fine but I
guess the DDBB reach a certain volume that made it fail.

I have read that it is related to global max_allowed_packet which has
to be set to a higher value, with the following expression

set global max_allowed_packet = 1073741824;

I have my ODK aggregate deployed in a google app engine, so i tied to
type that in the cloud shell but nothing happened, I still have the same
problem, and as you can imaging it´s the worst moment to happend...

I have a excel macro that reads the csv file and download the media
files, renames and save everything in a file, so then i can open each
picture from a link in an excel spreadsheet.

Now I can only download the the CSV using ODK briefcase by:
1- Pulling all the data.
2- Exporting the CSV (I can only do it if I firstly pull the data)

The problem is that by doing this I have to download the media files
twice (in briefcase and in my macro), and the CSV file exported by
briefcase contains the local link in my computer to the media files, but
not the cloud link to the media files, which in my case it is super
convenient. I figure out that the link to the media is a concatenation of
the ID number and some other things, so I manage to bypass this problem,
but I had to change my whole procedure in the middle of the work, making it
much more manual and not allowing me to use the tools I developped to
automate all this work.

Anyone knows how can I in ODK aggregate deployed in google app engine
execute this procedure?
set global max_allowed_packet = 1073741824;

IF that really is the problem, which i hope it is.

Thank you very much in advance.

Regards,

Pablo

--

Post: opendatakit@googlegroups.com
Unsubscribe: opendatakit+unsubscribe@googlegroups.com
Options: http://groups.google.com/group/opendatakit?hl=en

You received this message because you are subscribed to the Google
Groups "ODK Community" group.
To unsubscribe from this group and stop receiving emails from it, send
an email to opendatakit+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--

Post: opendatakit@googlegroups.com
Unsubscribe: opendatakit+unsubscribe@googlegroups.com
Options: http://groups.google.com/group/opendatakit?hl=en

You received this message because you are subscribed to the Google Groups
"ODK Community" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to opendatakit+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--
Mitch Sundt
Software Engineer
University of Washington
mitchellsundt@gmail.com

Pablo_Rodriguez · May 6, 2016, 10:27am

It is more likely that you are running out of memory on the ODK Aggregate server.

When you Export to CSV, the entire dataset must be held in memory while the CSV is generated. At some point that fails, and you need to move onto either ODK Briefcase and its export functionality or to a publisher.

You can change the size of your servers by editing the files within the ODKAggregate directories (this assumes you have upgraded to ODK Aggregate 1.4.8 -- you cannot do this on earlier versions).

To change the size of the server that affects the Export to CSV feature, you must edit:

ODKAggregate\background\WEB-INF\appengine-web.xml

(using Notepad -- not Word)

Then change the B2 line.

You want to change B2 to a larger instance class.

Google (apparently) no longer wants to provide documentation on what their class names mean.

The available classes are:

B1, B2, B4, B4_1G, and B8

If memory serves, these roughly correspond to 128MB, 256MB, 512MB, 1GB and 1GB of available memory.

So if you change from B2 to B4 (and re-upload your ODK Aggregate configuration), you will double your JVM size, which will allow you to emit 2x more data via CSV.

Thank you Yaw, I´m publising to google spreadsheets, because I am not familiar with fusion tables, it´s going slow, but it´s going.

Thanks,

Pablo

Pablo,

max_allowed_packet is a MySQL command and so only works on self-hosted

solutions backed my MySQL. Since you are on Google App Engine, that

command will not work.

One easy to thing to try is to stream your data to Google FusionTables

or Sheets and then download the CSV those systems generate.

Yaw

--

Need ODK consultants? Nafundi provides form design, server setup,

in-field training, and software development for ODK. Go to

https://nafundi.com to get started.

Hi guys,

I´m kind of new here, but I already have this problem:

On ODK Aggregate, when I want to export my data to .csv file, It´s showing me an error (Failure - will retry later). it was working fine but I guess the DDBB reach a certain volume that made it fail.

I have read that it is related to global max_allowed_packet which has to be set to a higher value, with the following expression

set global max_allowed_packet = 1073741824;

I have my ODK aggregate deployed in a google app engine, so i tied to type that in the cloud shell but nothing happened, I still have the same problem, and as you can imaging it´s the worst moment to happend...

I have a excel macro that reads the csv file and download the media files, renames and save everything in a file, so then i can open each picture from a link in an excel spreadsheet.

Now I can only download the the CSV using ODK briefcase by:

1- Pulling all the data.

2- Exporting the CSV (I can only do it if I firstly pull the data)

The problem is that by doing this I have to download the media files twice (in briefcase and in my macro), and the CSV file exported by briefcase contains the local link in my computer to the media files, but not the cloud link to the media files, which in my case it is super convenient. I figure out that the link to the media is a concatenation of the ID number and some other things, so I manage to bypass this problem, but I had to change my whole procedure in the middle of the work, making it much more manual and not allowing me to use the tools I developped to automate all this work.

Anyone knows how can I in ODK aggregate deployed in google app engine execute this procedure?

set global max_allowed_packet = 1073741824;

IF that really is the problem, which i hope it is.

Thank you very much in advance.

Regards,

Pablo

--

--

Post: opend...@googlegroups.com

Unsubscribe: opendatakit...@googlegroups.com

Options: http://groups.google.com/group/opendatakit?hl=en

You received this message because you are subscribed to the Google Groups "ODK Community" group.

To unsubscribe from this group and stop receiving emails from it, send an email to opendatakit...@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--

--

Post: opend...@googlegroups.com

Unsubscribe: opendatakit...@googlegroups.com

Options: http://groups.google.com/group/opendatakit?hl=en

You received this message because you are subscribed to the Google Groups "ODK Community" group.

To unsubscribe from this group and stop receiving emails from it, send an email to opendatakit...@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--

Mitch Sundt
Software Engineer
University of Washington
mitche...@gmail.com

Hi Mitch,

I think you are right, because when trying to export to a google spreadsheet it fail at entry 2600 something... and I have more than 3000 thousands. The problem is right now we are in a current project and I guess that for upgrading I´ll have to pull everything with briefcase, delete the project and deployed the ODK aggregate new version with the changes you tell me in google app engine. then once i have the project running again Íll have to push all the information I pulled with briefcase from the previous version of ODK. So as you can imaging is not the best moment to do it.

It is that correct?

Thank you very much for your support.

Regards,

Pablo

···

El martes, 3 de mayo de 2016, 20:18:14 (UTC+2), Mitch Sundt escribió: > On Tue, May 3, 2016 at 7:53 AM, wrote: > El martes, 3 de mayo de 2016, 14:57:20 (UTC+2), Yaw Anokwa escribió: > > On Tue, May 3, 2016 at 2:10 PM, wrote:

Pablo_Rodriguez · May 6, 2016, 12:52pm

I am trying to pull all the information with briefcase, and looking at the log I have this:

...
fetching instance 3048 ...
fetching instance 3049 ...
fetching instance 3050 ...
fetching instance 3051 ...
fetching instance 3052 ...
fetching instance 3053 ...
retrieving next chunk of instances from server...
FAILED.

I don´t know if it failed because there is nothing more to pull or because actually there is something elses happening... right now I don´t know how many entries I have so 3053? I don´t know... the filter in aggregate does not work so i don´t know how could I do this...

···

El viernes, 6 de mayo de 2016, 12:30:39 (UTC+2), nuncaestardes...@gmail.com escribió: > El martes, 3 de mayo de 2016, 14:10:28 (UTC+2), nuncaestardes...@gmail.com escribió: > > Hi guys, > > > > I´m kind of new here, but I already have this problem: > > > > On ODK Aggregate, when I want to export my data to .csv file, It´s showing me an error (Failure - will retry later). it was working fine but I guess the DDBB reach a certain volume that made it fail. > > > > I have read that it is related to global max_allowed_packet which has to be set to a higher value, with the following expression > > > > set global max_allowed_packet = 1073741824; > > > > I have my ODK aggregate deployed in a google app engine, so i tied to type that in the cloud shell but nothing happened, I still have the same problem, and as you can imaging it´s the worst moment to happend... > > > > I have a excel macro that reads the csv file and download the media files, renames and save everything in a file, so then i can open each picture from a link in an excel spreadsheet. > > > > Now I can only download the the CSV using ODK briefcase by: > > 1- Pulling all the data. > > 2- Exporting the CSV (I can only do it if I firstly pull the data) > > > > The problem is that by doing this I have to download the media files twice (in briefcase and in my macro), and the CSV file exported by briefcase contains the local link in my computer to the media files, but not the cloud link to the media files, which in my case it is super convenient. I figure out that the link to the media is a concatenation of the ID number and some other things, so I manage to bypass this problem, but I had to change my whole procedure in the middle of the work, making it much more manual and not allowing me to use the tools I developped to automate all this work. > > > > Anyone knows how can I in ODK aggregate deployed in google app engine execute this procedure? > > set global max_allowed_packet = 1073741824; > > > > IF that really is the problem, which i hope it is. > > > > Thank you very much in advance. > > > > Regards, > > > > Pablo > > One more thing, everytime I pull information with briefcase it downloads the whole project instead of just downloading the new entries,that could be something easy to improve in future versions, my excel macro that downloads everything from the csv does that it is muc more convenient this way. > > Regards, > > Pablo

Mitch_S · May 6, 2016, 4:22pm

This would indicate a data corruption issue.

See these instructions:

w.r.t. ODK Briefcase starting to pull data from the beginning every time,
if, after you pull data, you issue a push back up to the same server, this
will set tracking flags such that those records will be ignored and skipped
over the next time you pull.

ODK Briefcase stops at the first corrupted record because it is fetching
data in ascending marked-as-complete timestamp order. Until the corrupted
record is repaired or removed, it will not proceed to other records.

Data corruption is more likely to occur if your server memory is becoming
exhausted or if you are submitting data over very-low-bandwidth
communications channels (e.g., satellite).

···

On Fri, May 6, 2016 at 5:52 AM, wrote:

El viernes, 6 de mayo de 2016, 12:30:39 (UTC+2), nuncaestardes...@gmail.com escribió:

El martes, 3 de mayo de 2016, 14:10:28 (UTC+2), nuncaestardes...@gmail.com escribió:

Hi guys,

I´m kind of new here, but I already have this problem:

On ODK Aggregate, when I want to export my data to .csv file, It´s
showing me an error (Failure - will retry later). it was working fine but I
guess the DDBB reach a certain volume that made it fail.

I have read that it is related to global max_allowed_packet which has
to be set to a higher value, with the following expression

set global max_allowed_packet = 1073741824;

I have my ODK aggregate deployed in a google app engine, so i tied to
type that in the cloud shell but nothing happened, I still have the same
problem, and as you can imaging it´s the worst moment to happend...

I have a excel macro that reads the csv file and download the media
files, renames and save everything in a file, so then i can open each
picture from a link in an excel spreadsheet.

Now I can only download the the CSV using ODK briefcase by:
1- Pulling all the data.
2- Exporting the CSV (I can only do it if I firstly pull the data)

The problem is that by doing this I have to download the media files
twice (in briefcase and in my macro), and the CSV file exported by
briefcase contains the local link in my computer to the media files, but
not the cloud link to the media files, which in my case it is super
convenient. I figure out that the link to the media is a concatenation of
the ID number and some other things, so I manage to bypass this problem,
but I had to change my whole procedure in the middle of the work, making it
much more manual and not allowing me to use the tools I developped to
automate all this work.

Anyone knows how can I in ODK aggregate deployed in google app engine
execute this procedure?
set global max_allowed_packet = 1073741824;

IF that really is the problem, which i hope it is.

Thank you very much in advance.

Regards,

Pablo

One more thing, everytime I pull information with briefcase it downloads
the whole project instead of just downloading the new entries,that could be
something easy to improve in future versions, my excel macro that downloads
everything from the csv does that it is muc more convenient this way.

Regards,

Pablo

I am trying to pull all the information with briefcase, and looking at the
log I have this:

...
fetching instance 3048 ...
fetching instance 3049 ...
fetching instance 3050 ...
fetching instance 3051 ...
fetching instance 3052 ...
fetching instance 3053 ...
retrieving next chunk of instances from server...
FAILED.

I don´t know if it failed because there is nothing more to pull or because
actually there is something elses happening... right now I don´t know how
many entries I have so 3053? I don´t know... the filter in aggregate does
not work so i don´t know how could I do this...

--

Post: opendatakit@googlegroups.com
Unsubscribe: opendatakit+unsubscribe@googlegroups.com
Options: http://groups.google.com/group/opendatakit?hl=en

You received this message because you are subscribed to the Google Groups
"ODK Community" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to opendatakit+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--
Mitch Sundt
Software Engineer
University of Washington
mitchellsundt@gmail.com

Pablo_Rodriguez · June 6, 2016, 4:39pm

Well Now in another project I´m having the very same issue. wort timing
ever. In the middle of a big running project :S

There are not incomplete submissions according to ODK.

[image: Inline image 2]

I was publishing a google spreadsheet and a fusion table (Running in google
app engine with billing enabled)
[image: Inline image 3]

I´m looking at the log and I´m having an average of 4 to 7 error per
minute, basically like this:

18:14:55.688GET500956 B550 msAppEngine-Google; (+
http://code.google.com/appengine)
/gae/uploadSubmissionsTask?fscUri=uuid%3A9672732c-1bc6-4dde-a463-d066b5f3edf3
0.1.0.2 - - [06/Jun/2016:18:14:55 +0200] "GET
/gae/uploadSubmissionsTask?fscUri=uuid%3A9672732c-1bc6-4dde-a463-d066b5f3edf3
HTTP/1.1" 500 956
http://olmedo-servicios-integrales.appspot.com/gae/watchdog
"AppEngine-Google; (+http://code.google.com/appengine)" "
olmedo-servicios-integrales.appspot.com" ms=550 cpu_ms=231
cpm_usd=2.5548e-7 loading_request=0
instance=00c61b117c327d751511c82e8ba427e6081789412cae8a6b2a33f690566e8b31
app_engine_release=1.9.38 trace_id=-
{
metadata: {…}
protoPayload: {…}
insertId: "2016-06-06|09:15:00.884913-07|10.106.162.86|1409825575"
log: "appengine.googleapis.com/request_log"
httpRequest: {…}
operation: {…}
}

Seaching for OutOfMemory in the log and I don´t find anything.

Right now I just pulled all the information with Briefcase, and As far as I
can see I have more than what I had in my published tables.

My biggest concern is that I could loose submissions :S.

Any help is very appreciated.

Regards,

···

On Fri, May 6, 2016 at 6:22 PM, Mitch Sundt wrote:

This would indicate a data corruption issue.

See these instructions:
https://github.com/opendatakit/opendatakit/wiki/Aggregate-AppEngine-Troubleshooting#reparing-a-filled-in-form-submission

w.r.t. ODK Briefcase starting to pull data from the beginning every time,
if, after you pull data, you issue a push back up to the same server, this
will set tracking flags such that those records will be ignored and skipped
over the next time you pull.

ODK Briefcase stops at the first corrupted record because it is fetching
data in ascending marked-as-complete timestamp order. Until the corrupted
record is repaired or removed, it will not proceed to other records.

Data corruption is more likely to occur if your server memory is becoming
exhausted or if you are submitting data over very-low-bandwidth
communications channels (e.g., satellite).

On Fri, May 6, 2016 at 5:52 AM, nuncaestardesillegaspronto@gmail.com wrote:

El viernes, 6 de mayo de 2016, 12:30:39 (UTC+2), nuncaestardes...@gmail.com escribió:

El martes, 3 de mayo de 2016, 14:10:28 (UTC+2), nuncaestardes...@gmail.com escribió:

Hi guys,

I´m kind of new here, but I already have this problem:

On ODK Aggregate, when I want to export my data to .csv file, It´s
showing me an error (Failure - will retry later). it was working fine but I
guess the DDBB reach a certain volume that made it fail.

I have read that it is related to global max_allowed_packet which has
to be set to a higher value, with the following expression

set global max_allowed_packet = 1073741824;

I have my ODK aggregate deployed in a google app engine, so i tied to
type that in the cloud shell but nothing happened, I still have the same
problem, and as you can imaging it´s the worst moment to happend...

I have a excel macro that reads the csv file and download the media
files, renames and save everything in a file, so then i can open each
picture from a link in an excel spreadsheet.

Now I can only download the the CSV using ODK briefcase by:
1- Pulling all the data.
2- Exporting the CSV (I can only do it if I firstly pull the data)

The problem is that by doing this I have to download the media files
twice (in briefcase and in my macro), and the CSV file exported by
briefcase contains the local link in my computer to the media files, but
not the cloud link to the media files, which in my case it is super
convenient. I figure out that the link to the media is a concatenation of
the ID number and some other things, so I manage to bypass this problem,
but I had to change my whole procedure in the middle of the work, making it
much more manual and not allowing me to use the tools I developped to
automate all this work.

Anyone knows how can I in ODK aggregate deployed in google app engine
execute this procedure?
set global max_allowed_packet = 1073741824;

IF that really is the problem, which i hope it is.

Thank you very much in advance.

Regards,

Pablo

One more thing, everytime I pull information with briefcase it
downloads the whole project instead of just downloading the new
entries,that could be something easy to improve in future versions, my
excel macro that downloads everything from the csv does that it is muc more
convenient this way.

Regards,

Pablo

I am trying to pull all the information with briefcase, and looking at
the log I have this:

...
fetching instance 3048 ...
fetching instance 3049 ...
fetching instance 3050 ...
fetching instance 3051 ...
fetching instance 3052 ...
fetching instance 3053 ...
retrieving next chunk of instances from server...
FAILED.

I don´t know if it failed because there is nothing more to pull or
because actually there is something elses happening... right now I don´t
know how many entries I have so 3053? I don´t know... the filter in
aggregate does not work so i don´t know how could I do this...

--

Post: opendatakit@googlegroups.com
Unsubscribe: opendatakit+unsubscribe@googlegroups.com
Options: http://groups.google.com/group/opendatakit?hl=en

You received this message because you are subscribed to the Google Groups
"ODK Community" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to opendatakit+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--
Mitch Sundt
Software Engineer
University of Washington
mitchellsundt@gmail.com

--

Post: opendatakit@googlegroups.com
Unsubscribe: opendatakit+unsubscribe@googlegroups.com
Options: http://groups.google.com/group/opendatakit?hl=en

You received this message because you are subscribed to a topic in the
Google Groups "ODK Community" group.
To unsubscribe from this topic, visit
https://groups.google.com/d/topic/opendatakit/hjMKcHkk1tc/unsubscribe.
To unsubscribe from this group and all its topics, send an email to
opendatakit+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Mitch_S · June 6, 2016, 10:55pm

Have you worked through the database repair steps described here:

I.e.,

org.opendatakit.aggregate.task.gae.servlet.UploadSubmissionsTaskServlet
doGet: org.opendatakit.aggregate.exception.ODKExternalServiceException:
org.opendatakit.common.datamodel.ODKEnumeratedElementException: SELECT *
FROM MAPFRE_JUNIO2016_G3_REPORTE WHERE _TOP_LEVEL_AURI =
uuid:9e03e883-84bd-4457-a9cd-ae54746e1139 AND _PARENT_AURI =
uuid:9e03e883-84bd-4457-a9cd-ae54746e1139 multiple-choice valueSet is
missing an entry OR has an extra copy of one

and then, on the datastore tab,

entering in the query box:

SELECT * FROM opendatakit.MAPFRE_JUNIO2016_G3_REPORTE WHERE
_TOP_LEVEL_AURI = "uuid:9e03e883-84bd-4457-a9cd-ae54746e1139" AND
_PARENT_AURI = "uuid:9e03e883-84bd-4457-a9cd-ae54746e1139"

you'll find 2 entries with matching _ORDINAL_NUMBER values (1).

Per the instructions (step 6 & 7), delete the older one of these.

···

On Mon, Jun 6, 2016 at 9:39 AM, Pablo Rodríguez < nuncaestardesillegaspronto@gmail.com> wrote:

Well Now in another project I´m having the very same issue. wort timing
ever. In the middle of a big running project :S

There are not incomplete submissions according to ODK.

[image: Inline image 2]

I was publishing a google spreadsheet and a fusion table (Running in
google app engine with billing enabled)
[image: Inline image 3]

I´m looking at the log and I´m having an average of 4 to 7 error per
minute, basically like this:

18:14:55.688GET500956 B550 msAppEngine-Google; (+
http://code.google.com/appengine)
/gae/uploadSubmissionsTask?fscUri=uuid%3A9672732c-1bc6-4dde-a463-d066b5f3edf3
0.1.0.2 - - [06/Jun/2016:18:14:55 +0200] "GET
/gae/uploadSubmissionsTask?fscUri=uuid%3A9672732c-1bc6-4dde-a463-d066b5f3edf3
HTTP/1.1" 500 956
http://olmedo-servicios-integrales.appspot.com/gae/watchdog
"AppEngine-Google; (+http://code.google.com/appengine)" "
olmedo-servicios-integrales.appspot.com" ms=550 cpu_ms=231
cpm_usd=2.5548e-7 loading_request=0
instance=00c61b117c327d751511c82e8ba427e6081789412cae8a6b2a33f690566e8b31
app_engine_release=1.9.38 trace_id=-
{
metadata: {…}
protoPayload: {…}
insertId: "2016-06-06|09:15:00.884913-07|10.106.162.86|1409825575"
log: "appengine.googleapis.com/request_log"
httpRequest: {…}
operation: {…}
}

Seaching for OutOfMemory in the log and I don´t find anything.

Right now I just pulled all the information with Briefcase, and As far as
I can see I have more than what I had in my published tables.

My biggest concern is that I could loose submissions :S.

Any help is very appreciated.

Regards,

On Fri, May 6, 2016 at 6:22 PM, Mitch Sundt mitchellsundt@gmail.com wrote:

This would indicate a data corruption issue.

See these instructions:
https://github.com/opendatakit/opendatakit/wiki/Aggregate-AppEngine-Troubleshooting#reparing-a-filled-in-form-submission

w.r.t. ODK Briefcase starting to pull data from the beginning every time,
if, after you pull data, you issue a push back up to the same server, this
will set tracking flags such that those records will be ignored and skipped
over the next time you pull.

ODK Briefcase stops at the first corrupted record because it is fetching
data in ascending marked-as-complete timestamp order. Until the corrupted
record is repaired or removed, it will not proceed to other records.

Data corruption is more likely to occur if your server memory is becoming
exhausted or if you are submitting data over very-low-bandwidth
communications channels (e.g., satellite).

On Fri, May 6, 2016 at 5:52 AM, nuncaestardesillegaspronto@gmail.com wrote:

El viernes, 6 de mayo de 2016, 12:30:39 (UTC+2), nuncaestardes...@gmail.com escribió:

El martes, 3 de mayo de 2016, 14:10:28 (UTC+2), nuncaestardes...@gmail.com escribió:

Hi guys,

I´m kind of new here, but I already have this problem:

On ODK Aggregate, when I want to export my data to .csv file, It´s
showing me an error (Failure - will retry later). it was working fine but I
guess the DDBB reach a certain volume that made it fail.

I have read that it is related to global max_allowed_packet which
has to be set to a higher value, with the following expression

set global max_allowed_packet = 1073741824;

I have my ODK aggregate deployed in a google app engine, so i tied
to type that in the cloud shell but nothing happened, I still have the same
problem, and as you can imaging it´s the worst moment to happend...

I have a excel macro that reads the csv file and download the media
files, renames and save everything in a file, so then i can open each
picture from a link in an excel spreadsheet.

Now I can only download the the CSV using ODK briefcase by:
1- Pulling all the data.
2- Exporting the CSV (I can only do it if I firstly pull the data)

The problem is that by doing this I have to download the media files
twice (in briefcase and in my macro), and the CSV file exported by
briefcase contains the local link in my computer to the media files, but
not the cloud link to the media files, which in my case it is super
convenient. I figure out that the link to the media is a concatenation of
the ID number and some other things, so I manage to bypass this problem,
but I had to change my whole procedure in the middle of the work, making it
much more manual and not allowing me to use the tools I developped to
automate all this work.

Anyone knows how can I in ODK aggregate deployed in google app
engine execute this procedure?
set global max_allowed_packet = 1073741824;

IF that really is the problem, which i hope it is.

Thank you very much in advance.

Regards,

Pablo

One more thing, everytime I pull information with briefcase it
downloads the whole project instead of just downloading the new
entries,that could be something easy to improve in future versions, my
excel macro that downloads everything from the csv does that it is muc more
convenient this way.

Regards,

Pablo

I am trying to pull all the information with briefcase, and looking at
the log I have this:

...
fetching instance 3048 ...
fetching instance 3049 ...
fetching instance 3050 ...
fetching instance 3051 ...
fetching instance 3052 ...
fetching instance 3053 ...
retrieving next chunk of instances from server...
FAILED.

I don´t know if it failed because there is nothing more to pull or
because actually there is something elses happening... right now I don´t
know how many entries I have so 3053? I don´t know... the filter in
aggregate does not work so i don´t know how could I do this...

--

Post: opendatakit@googlegroups.com
Unsubscribe: opendatakit+unsubscribe@googlegroups.com
Options: http://groups.google.com/group/opendatakit?hl=en

You received this message because you are subscribed to the Google
Groups "ODK Community" group.
To unsubscribe from this group and stop receiving emails from it, send
an email to opendatakit+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--
Mitch Sundt
Software Engineer
University of Washington
mitchellsundt@gmail.com

--

Post: opendatakit@googlegroups.com
Unsubscribe: opendatakit+unsubscribe@googlegroups.com
Options: http://groups.google.com/group/opendatakit?hl=en

You received this message because you are subscribed to a topic in the
Google Groups "ODK Community" group.
To unsubscribe from this topic, visit
https://groups.google.com/d/topic/opendatakit/hjMKcHkk1tc/unsubscribe.
To unsubscribe from this group and all its topics, send an email to
opendatakit+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--

Post: opendatakit@googlegroups.com
Unsubscribe: opendatakit+unsubscribe@googlegroups.com
Options: http://groups.google.com/group/opendatakit?hl=en

You received this message because you are subscribed to the Google Groups
"ODK Community" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to opendatakit+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--
Mitch Sundt
Software Engineer
University of Washington
mitchellsundt@gmail.com

Pablo_Rodriguez · June 7, 2016, 7:22am

I´m going to try Mitch, for me this is dancing on thin ice... let´s see...!

Thanks!

···

El martes, 7 de junio de 2016, 0:55:47 (UTC+2), Mitch Sundt escribió: > Have you worked through the database repair steps described here: > > > https://github.com/opendatakit/opendatakit/wiki/Aggregate-AppEngine-Troubleshooting#repairing-a-filled-in-form-submission > > > I.e., > > > org.opendatakit.aggregate.task.gae.servlet.UploadSubmissionsTaskServlet doGet: org.opendatakit.aggregate.exception.ODKExternalServiceException: org.opendatakit.common.datamodel.ODKEnumeratedElementException: SELECT * FROM MAPFRE_JUNIO2016_G3_REPORTE WHERE _TOP_LEVEL_AURI = uuid:9e03e883-84bd-4457-a9cd-ae54746e1139 AND _PARENT_AURI = uuid:9e03e883-84bd-4457-a9cd-ae54746e1139 multiple-choice valueSet is missing an entry OR has an extra copy of one > > > > > and then, on the datastore tab, > > > entering in the query box: > > > > SELECT * FROM `opendatakit.MAPFRE_JUNIO2016_G3_REPORTE` WHERE _TOP_LEVEL_AURI = "uuid:9e03e883-84bd-4457-a9cd-ae54746e1139" AND _PARENT_AURI = "uuid:9e03e883-84bd-4457-a9cd-ae54746e1139" > > > you'll find 2 entries with matching _ORDINAL_NUMBER values (1). > > > Per the instructions (step 6 & 7), delete the older one of these. > > > > > On Mon, Jun 6, 2016 at 9:39 AM, Pablo Rodríguez wrote: > > Well Now in another project I´m having the very same issue. wort timing ever. In the middle of a big running project :S > > > There are not incomplete submissions according to ODK. > > > > > > > I was publishing a google spreadsheet and a fusion table (Running in google app engine with billing enabled) > > > > > I´m looking at the log and I´m having an average of 4 to 7 error per minute, basically like this: > > > > > > 18:14:55.688GET500956 B550 msAppEngine-Google; (+http://code.google.com/appengine)/gae/uploadSubmissionsTask?fscUri=uuid%3A9672732c-1bc6-4dde-a463-d066b5f3edf3 > > > 0.1.0.2 - - [06/Jun/2016:18:14:55 +0200] "GET /gae/uploadSubmissionsTask?fscUri=uuid%3A9672732c-1bc6-4dde-a463-d066b5f3edf3 HTTP/1.1" 500 956 http://olmedo-servicios-integrales.appspot.com/gae/watchdog "AppEngine-Google; (+http://code.google.com/appengine)" "olmedo-servicios-integrales.appspot.com" ms=550 cpu_ms=231 cpm_usd=2.5548e-7 loading_request=0 instance=00c61b117c327d751511c82e8ba427e6081789412cae8a6b2a33f690566e8b31 app_engine_release=1.9.38 trace_id=- > > > { > > metadata: {…} > protoPayload: {…} > insertId: "2016-06-06|09:15:00.884913-07|10.106.162.86|1409825575" > log: "appengine.googleapis.com/request_log" > httpRequest: {…} > operation: {…} } > > > Seaching for OutOfMemory in the log and I don´t find anything. > > > Right now I just pulled all the information with Briefcase, and As far as I can see I have more than what I had in my published tables. > > > My biggest concern is that I could loose submissions :S. > > > Any help is very appreciated. > > > Regards, > > > > > On Fri, May 6, 2016 at 6:22 PM, Mitch Sundt wrote: > > > > This would indicate a data corruption issue. > > > See these instructions: https://github.com/opendatakit/opendatakit/wiki/Aggregate-AppEngine-Troubleshooting#reparing-a-filled-in-form-submission > > > w.r.t. ODK Briefcase starting to pull data from the beginning every time, if, after you pull data, you issue a push back up to the same server, this will set tracking flags such that those records will be ignored and skipped over the next time you pull. > > > ODK Briefcase stops at the first corrupted record because it is fetching data in ascending marked-as-complete timestamp order. Until the corrupted record is repaired or removed, it will not proceed to other records. > > > Data corruption is more likely to occur if your server memory is becoming exhausted or if you are submitting data over very-low-bandwidth communications channels (e.g., satellite). > > > > > > > On Fri, May 6, 2016 at 5:52 AM, wrote: > > > El viernes, 6 de mayo de 2016, 12:30:39 (UTC+2), nuncaestardes...@gmail.com escribió: > > > El martes, 3 de mayo de 2016, 14:10:28 (UTC+2), nuncaestardes...@gmail.com escribió: > > > > Hi guys, > > > > > > > > I´m kind of new here, but I already have this problem: > > > > > > > > On ODK Aggregate, when I want to export my data to .csv file, It´s showing me an error (Failure - will retry later). it was working fine but I guess the DDBB reach a certain volume that made it fail. > > > > > > > > I have read that it is related to global max_allowed_packet which has to be set to a higher value, with the following expression > > > > > > > > set global max_allowed_packet = 1073741824; > > > > > > > > I have my ODK aggregate deployed in a google app engine, so i tied to type that in the cloud shell but nothing happened, I still have the same problem, and as you can imaging it´s the worst moment to happend... > > > > > > > > I have a excel macro that reads the csv file and download the media files, renames and save everything in a file, so then i can open each picture from a link in an excel spreadsheet. > > > > > > > > Now I can only download the the CSV using ODK briefcase by: > > > > 1- Pulling all the data. > > > > 2- Exporting the CSV (I can only do it if I firstly pull the data) > > > > > > > > The problem is that by doing this I have to download the media files twice (in briefcase and in my macro), and the CSV file exported by briefcase contains the local link in my computer to the media files, but not the cloud link to the media files, which in my case it is super convenient. I figure out that the link to the media is a concatenation of the ID number and some other things, so I manage to bypass this problem, but I had to change my whole procedure in the middle of the work, making it much more manual and not allowing me to use the tools I developped to automate all this work. > > > > > > > > Anyone knows how can I in ODK aggregate deployed in google app engine execute this procedure? > > > > set global max_allowed_packet = 1073741824; > > > > > > > > IF that really is the problem, which i hope it is. > > > > > > > > Thank you very much in advance. > > > > > > > > Regards, > > > > > > > > Pablo > > > > > > One more thing, everytime I pull information with briefcase it downloads the whole project instead of just downloading the new entries,that could be something easy to improve in future versions, my excel macro that downloads everything from the csv does that it is muc more convenient this way. > > > > > > Regards, > > > > > > Pablo > > > > I am trying to pull all the information with briefcase, and looking at the log I have this: > > > > ... > > fetching instance 3048 ... > > fetching instance 3049 ... > > fetching instance 3050 ... > > fetching instance 3051 ... > > fetching instance 3052 ... > > fetching instance 3053 ... > > retrieving next chunk of instances from server... > > FAILED. > > > > I don´t know if it failed because there is nothing more to pull or because actually there is something elses happening... right now I don´t know how many entries I have so 3053? I don´t know... the filter in aggregate does not work so i don´t know how could I do this... > > > > > > -- > > -- > > Post: opend...@googlegroups.com > > Unsubscribe: opendatakit...@googlegroups.com > > Options: http://groups.google.com/group/opendatakit?hl=en > > > > --- > > You received this message because you are subscribed to the Google Groups "ODK Community" group. > > To unsubscribe from this group and stop receiving emails from it, send an email to opendatakit...@googlegroups.com. > > For more options, visit https://groups.google.com/d/optout. > > > > > > -- > > Mitch Sundt > Software Engineer > University of Washington > mitche...@gmail.com > > > > > > > > > > -- > > -- > > Post: opend...@googlegroups.com > > Unsubscribe: opendatakit...@googlegroups.com > > Options: http://groups.google.com/group/opendatakit?hl=en > > > > --- > > You received this message because you are subscribed to a topic in the Google Groups "ODK Community" group. > > To unsubscribe from this topic, visit https://groups.google.com/d/topic/opendatakit/hjMKcHkk1tc/unsubscribe. > > To unsubscribe from this group and all its topics, send an email to opendatakit...@googlegroups.com. > > For more options, visit https://groups.google.com/d/optout. > > > > > > > > > > -- > > -- > > Post: opend...@googlegroups.com > > Unsubscribe: opendatakit...@googlegroups.com > > Options: http://groups.google.com/group/opendatakit?hl=en > > > > --- > > You received this message because you are subscribed to the Google Groups "ODK Community" group. > > To unsubscribe from this group and stop receiving emails from it, send an email to opendatakit...@googlegroups.com. > > For more options, visit https://groups.google.com/d/optout. > > > > > > -- > > Mitch Sundt > Software Engineer > University of Washington > mitche...@gmail.com

Pablo_Rodriguez · June 7, 2016, 7:51am

I´ve done exactly what do told me, now I´m pulling with briefcase to see if
it works, but ends with FAILED
[image: Inline image 1]

The problem is that I´m struggling to understand what I am doing, so the
steps for Repairing a Filled-in Form Submission are not straight forward
for me. I´ll keep trying...

···

On Tue, Jun 7, 2016 at 9:22 AM, wrote:

I´m going to try Mitch, for me this is dancing on thin ice... let´s see...!

Thanks!

El martes, 7 de junio de 2016, 0:55:47 (UTC+2), Mitch Sundt escribió:

Have you worked through the database repair steps described here:

https://github.com/opendatakit/opendatakit/wiki/Aggregate-AppEngine-Troubleshooting#repairing-a-filled-in-form-submission

I.e.,

org.opendatakit.aggregate.task.gae.servlet.UploadSubmissionsTaskServlet
doGet: org.opendatakit.aggregate.exception.ODKExternalServiceException:
org.opendatakit.common.datamodel.ODKEnumeratedElementException: SELECT *
FROM MAPFRE_JUNIO2016_G3_REPORTE WHERE _TOP_LEVEL_AURI =
uuid:9e03e883-84bd-4457-a9cd-ae54746e1139 AND _PARENT_AURI =
uuid:9e03e883-84bd-4457-a9cd-ae54746e1139 multiple-choice valueSet is
missing an entry OR has an extra copy of one

and then, on the datastore tab,

entering in the query box:

SELECT * FROM opendatakit.MAPFRE_JUNIO2016_G3_REPORTE WHERE
_TOP_LEVEL_AURI = "uuid:9e03e883-84bd-4457-a9cd-ae54746e1139" AND
_PARENT_AURI = "uuid:9e03e883-84bd-4457-a9cd-ae54746e1139"

you'll find 2 entries with matching _ORDINAL_NUMBER values (1).

Per the instructions (step 6 & 7), delete the older one of these.

On Mon, Jun 6, 2016 at 9:39 AM, Pablo Rodríguez < nuncaestardes...@gmail.com> wrote:

Well Now in another project I´m having the very same issue. wort timing
ever. In the middle of a big running project :S

There are not incomplete submissions according to ODK.

I was publishing a google spreadsheet and a fusion table (Running in
google app engine with billing enabled)

I´m looking at the log and I´m having an average of 4 to 7 error per
minute, basically like this:

18:14:55.688GET500956 B550 msAppEngine-Google; (+
http://code.google.com/appengine)/gae/uploadSubmissionsTask?fscUri=uuid%3A9672732c-1bc6-4dde-a463-d066b5f3edf3

0.1.0.2 - - [06/Jun/2016:18:14:55 +0200] "GET
/gae/uploadSubmissionsTask?fscUri=uuid%3A9672732c-1bc6-4dde-a463-d066b5f3edf3
HTTP/1.1" 500 956
http://olmedo-servicios-integrales.appspot.com/gae/watchdog
"AppEngine-Google; (+http://code.google.com/appengine)" "
olmedo-servicios-integrales.appspot.com" ms=550 cpu_ms=231
cpm_usd=2.5548e-7 loading_request=0
instance=00c61b117c327d751511c82e8ba427e6081789412cae8a6b2a33f690566e8b31
app_engine_release=1.9.38 trace_id=-

{

metadata: {…}
protoPayload: {…}
insertId: "2016-06-06|09:15:00.884913-07|10.106.162.86|1409825575"
log: "appengine.googleapis.com/request_log"
httpRequest: {…}
operation: {…} }

Seaching for OutOfMemory in the log and I don´t find anything.

Right now I just pulled all the information with Briefcase, and As far
as I can see I have more than what I had in my published tables.

My biggest concern is that I could loose submissions :S.

Any help is very appreciated.

Regards,

On Fri, May 6, 2016 at 6:22 PM, Mitch Sundt mitche...@gmail.com wrote:

This would indicate a data corruption issue.

See these instructions:
https://github.com/opendatakit/opendatakit/wiki/Aggregate-AppEngine-Troubleshooting#reparing-a-filled-in-form-submission

w.r.t. ODK Briefcase starting to pull data from the beginning every
time, if, after you pull data, you issue a push back up to the same server,
this will set tracking flags such that those records will be ignored and
skipped over the next time you pull.

ODK Briefcase stops at the first corrupted record because it is fetching
data in ascending marked-as-complete timestamp order. Until the corrupted
record is repaired or removed, it will not proceed to other records.

Data corruption is more likely to occur if your server memory is
becoming exhausted or if you are submitting data over very-low-bandwidth
communications channels (e.g., satellite).

On Fri, May 6, 2016 at 5:52 AM, nuncaestardes...@gmail.com wrote:

El viernes, 6 de mayo de 2016, 12:30:39 (UTC+2), nuncaestardes...@gmail.com escribió:

El martes, 3 de mayo de 2016, 14:10:28 (UTC+2), nuncaestardes...@gmail.com escribió:

Hi guys,

I´m kind of new here, but I already have this problem:

On ODK Aggregate, when I want to export my data to .csv file, It´s
showing me an error (Failure - will retry later). it was working fine but I
guess the DDBB reach a certain volume that made it fail.

I have read that it is related to global max_allowed_packet which
has to be set to a higher value, with the following expression

set global max_allowed_packet = 1073741824;

I have my ODK aggregate deployed in a google app engine, so i tied
to type that in the cloud shell but nothing happened, I still have the same
problem, and as you can imaging it´s the worst moment to happend...

I have a excel macro that reads the csv file and download the media
files, renames and save everything in a file, so then i can open each
picture from a link in an excel spreadsheet.

Now I can only download the the CSV using ODK briefcase by:

1- Pulling all the data.

2- Exporting the CSV (I can only do it if I firstly pull the data)

The problem is that by doing this I have to download the media files
twice (in briefcase and in my macro), and the CSV file exported by
briefcase contains the local link in my computer to the media files, but
not the cloud link to the media files, which in my case it is super
convenient. I figure out that the link to the media is a concatenation of
the ID number and some other things, so I manage to bypass this problem,
but I had to change my whole procedure in the middle of the work, making it
much more manual and not allowing me to use the tools I developped to
automate all this work.

Anyone knows how can I in ODK aggregate deployed in google app
engine execute this procedure?

set global max_allowed_packet = 1073741824;

IF that really is the problem, which i hope it is.

Thank you very much in advance.

Regards,

Pablo

One more thing, everytime I pull information with briefcase it
downloads the whole project instead of just downloading the new
entries,that could be something easy to improve in future versions, my
excel macro that downloads everything from the csv does that it is muc more
convenient this way.

Regards,

Pablo

I am trying to pull all the information with briefcase, and looking at
the log I have this:

...

fetching instance 3048 ...

fetching instance 3049 ...

fetching instance 3050 ...

fetching instance 3051 ...

fetching instance 3052 ...

fetching instance 3053 ...

retrieving next chunk of instances from server...

FAILED.

I don´t know if it failed because there is nothing more to pull or
because actually there is something elses happening... right now I don´t
know how many entries I have so 3053? I don´t know... the filter in
aggregate does not work so i don´t know how could I do this...

--

--

Post: opend...@googlegroups.com

Unsubscribe: opendatakit...@googlegroups.com

Options: http://groups.google.com/group/opendatakit?hl=en

You received this message because you are subscribed to the Google
Groups "ODK Community" group.

To unsubscribe from this group and stop receiving emails from it, send
an email to opendatakit...@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--

Mitch Sundt
Software Engineer
University of Washington
mitche...@gmail.com

--

--

Post: opend...@googlegroups.com

Unsubscribe: opendatakit...@googlegroups.com

Options: http://groups.google.com/group/opendatakit?hl=en

You received this message because you are subscribed to a topic in the
Google Groups "ODK Community" group.

To unsubscribe from this topic, visit
https://groups.google.com/d/topic/opendatakit/hjMKcHkk1tc/unsubscribe.

To unsubscribe from this group and all its topics, send an email to
opendatakit...@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--

--

Post: opend...@googlegroups.com

Unsubscribe: opendatakit...@googlegroups.com

Options: http://groups.google.com/group/opendatakit?hl=en

You received this message because you are subscribed to the Google
Groups "ODK Community" group.

To unsubscribe from this group and stop receiving emails from it, send
an email to opendatakit...@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--

Mitch Sundt
Software Engineer
University of Washington
mitche...@gmail.com

--

Post: opendatakit@googlegroups.com
Unsubscribe: opendatakit+unsubscribe@googlegroups.com
Options: http://groups.google.com/group/opendatakit?hl=en

You received this message because you are subscribed to a topic in the
Google Groups "ODK Community" group.
To unsubscribe from this topic, visit
https://groups.google.com/d/topic/opendatakit/hjMKcHkk1tc/unsubscribe.
To unsubscribe from this group and all its topics, send an email to
opendatakit+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Pablo_Rodriguez · June 7, 2016, 8:09am

Ok So I see there are two errors constantly happening:
[image: Inline image 1]

one of them:

09:55:58.212GET500960 B5.9 sAppEngine-Google; (+
http://code.google.com/appengine)
/gae/uploadSubmissionsTask?fscUri=uuid%3A9672732c-1bc6-4dde-a463-d066b5f3edf3
0.1.0.2 - - [07/Jun/2016:09:55:58 +0200] "GET
/gae/uploadSubmissionsTask?fscUri=uuid%3A9672732c-1bc6-4dde-a463-d066b5f3edf3
HTTP/1.1" 500 960
http://olmedo-servicios-integrales.appspot.com/gae/watchdog
"AppEngine-Google; (+http://code.google.com/appengine)" "
olmedo-servicios-integrales.appspot.com" ms=5901 cpu_ms=460
cpm_usd=3.1940599999999997e-7 loading_request=0
instance=00c61b117c0e9e3d18748ca60cc00676c90a86e4344dfcefb7184ce68bcd8de904
app_engine_release=1.9.38 trace_id=-
{
metadata: {…}
protoPayload: {…}
insertId: "2016-06-07|00:56:08.408633-07|10.94.34.134|-1592936713"
log: "appengine.googleapis.com/request_log"
httpRequest: {…}
operation: {…}
}
09:56:01.221org.opendatakit.aggregate.task.gae.servlet.UploadSubmissionsTaskServlet
doGet: Beginning servlet processing
09:56:01.227org.opendatakit.aggregate.task.gae.servlet.UploadSubmissionsTaskServlet
doGet: Request is running on frontend
09:56:01.228org.opendatakit.aggregate.task.UploadSubmissionsWorkerImpl
uploadAllSubmissions: Beginning UPLOAD service:
uuid:8979e040-b496-49c1-b8de-f355d69d4c46 form MapfreJunio2016
09:56:01.228org.opendatakit.aggregate.form.FormFactory internalGetForms:
FormCache: using cached list of Forms
09:56:02.449org.opendatakit.aggregate.form.FormFactory internalGetForms:
FormCache: using cached list of Forms
09:56:02.478[s~olmedo-servicios-integrales/1.392851787586356529].:
Time left on lock: -73233127
09:56:02.478[s~olmedo-servicios-integrales/1.392851787586356529].:
Time left on lock: -13406599
09:56:02.478[s~olmedo-servicios-integrales/1.392851787586356529].:
Time left on lock: -52076838
09:56:02.478[s~olmedo-servicios-integrales/1.392851787586356529].:
Time left on lock: -57872843
09:56:02.478[s~olmedo-servicios-integrales/1.392851787586356529].:
Time left on lock: -78707824

And the other one

09:56:03.646GET2020 B1.4 sAppEngine-Google; (+
http://code.google.com/appengine)
/gae/uploadSubmissionsTask?fscUri=uuid%3A7ad28143-2a7a-4624-8807-215f05fd0102
0.1.0.2 - - [07/Jun/2016:09:56:03 +0200] "GET
/gae/uploadSubmissionsTask?fscUri=uuid%3A7ad28143-2a7a-4624-8807-215f05fd0102
HTTP/1.1" 202 - http://olmedo-servicios-integrales.appspot.com/gae/watchdog
"AppEngine-Google; (+http://code.google.com/appengine)" "
olmedo-servicios-integrales.appspot.com" ms=1400 cpu_ms=120
cpm_usd=1.39586e-7 loading_request=0
instance=00c61b117c0e9e3d18748ca60cc00676c90a86e4344dfcefb7184ce68bcd8de904
app_engine_release=1.9.38 trace_id=-
{
metadata: {…}
protoPayload: {…}
insertId: "2016-06-07|00:56:08.408751-07|10.94.34.134|-1592936713"
log: "appengine.googleapis.com/request_log"
httpRequest: {…}
operation: {…}
}
09:56:03.652org.opendatakit.aggregate.task.gae.servlet.UploadSubmissionsTaskServlet
doGet: Beginning servlet processing
09:56:03.658org.opendatakit.aggregate.task.gae.servlet.UploadSubmissionsTaskServlet
doGet: Request is running on frontend
09:56:03.658org.opendatakit.aggregate.task.UploadSubmissionsWorkerImpl
uploadAllSubmissions: Beginning UPLOAD service:
uuid:28dbf50b-2fcf-4424-800f-9c90eb823a09 form MapfreJunio2016
09:56:03.658org.opendatakit.aggregate.form.FormFactory internalGetForms:
FormCache: using cached list of Forms
09:56:03.856org.opendatakit.aggregate.form.FormFactory internalGetForms:
FormCache: using cached list of Forms
09:56:03.881[s~olmedo-servicios-integrales/1.392851787586356529].:
Time left on lock: -14278749
09:56:03.881[s~olmedo-servicios-integrales/1.392851787586356529].:
Time left on lock: -17503047
09:56:03.881[s~olmedo-servicios-integrales/1.392851787586356529].:
Time left on lock: -34874322
09:56:03.881[s~olmedo-servicios-integrales/1.392851787586356529].:
Time left on lock: -50316469
09:56:03.881[s~olmedo-servicios-integrales/1.392851787586356529].:
Time left on lock: -18963331

So I try to run your query but changing the uuid to see if i can see two
entries like the first thing I did but it says no entities matched. I
checked and the publishing is still not working, but when I pull with
Briefcase I can see it is downloading more instances, so I guess the system
keeps receiving forms...
[image: Inline image 2]

···

On Tue, Jun 7, 2016 at 9:51 AM, Pablo Rodríguez < nuncaestardesillegaspronto@gmail.com> wrote:

I´ve done exactly what do told me, now I´m pulling with briefcase to see
if it works, but ends with FAILED
[image: Inline image 1]

The problem is that I´m struggling to understand what I am doing, so the
steps for Repairing a Filled-in Form Submission are not straight forward
for me. I´ll keep trying...

On Tue, Jun 7, 2016 at 9:22 AM, nuncaestardesillegaspronto@gmail.com wrote:

I´m going to try Mitch, for me this is dancing on thin ice... let´s
see...!

Thanks!

El martes, 7 de junio de 2016, 0:55:47 (UTC+2), Mitch Sundt escribió:

Have you worked through the database repair steps described here:

https://github.com/opendatakit/opendatakit/wiki/Aggregate-AppEngine-Troubleshooting#repairing-a-filled-in-form-submission

I.e.,

org.opendatakit.aggregate.task.gae.servlet.UploadSubmissionsTaskServlet
doGet: org.opendatakit.aggregate.exception.ODKExternalServiceException:
org.opendatakit.common.datamodel.ODKEnumeratedElementException: SELECT *
FROM MAPFRE_JUNIO2016_G3_REPORTE WHERE _TOP_LEVEL_AURI =
uuid:9e03e883-84bd-4457-a9cd-ae54746e1139 AND _PARENT_AURI =
uuid:9e03e883-84bd-4457-a9cd-ae54746e1139 multiple-choice valueSet is
missing an entry OR has an extra copy of one

and then, on the datastore tab,

entering in the query box:

SELECT * FROM opendatakit.MAPFRE_JUNIO2016_G3_REPORTE WHERE
_TOP_LEVEL_AURI = "uuid:9e03e883-84bd-4457-a9cd-ae54746e1139" AND
_PARENT_AURI = "uuid:9e03e883-84bd-4457-a9cd-ae54746e1139"

you'll find 2 entries with matching _ORDINAL_NUMBER values (1).

Per the instructions (step 6 & 7), delete the older one of these.

On Mon, Jun 6, 2016 at 9:39 AM, Pablo Rodríguez < nuncaestardes...@gmail.com> wrote:

Well Now in another project I´m having the very same issue. wort timing
ever. In the middle of a big running project :S

There are not incomplete submissions according to ODK.

I was publishing a google spreadsheet and a fusion table (Running in
google app engine with billing enabled)

I´m looking at the log and I´m having an average of 4 to 7 error per
minute, basically like this:

18:14:55.688GET500956 B550 msAppEngine-Google; (+
http://code.google.com/appengine)/gae/uploadSubmissionsTask?fscUri=uuid%3A9672732c-1bc6-4dde-a463-d066b5f3edf3

0.1.0.2 - - [06/Jun/2016:18:14:55 +0200] "GET
/gae/uploadSubmissionsTask?fscUri=uuid%3A9672732c-1bc6-4dde-a463-d066b5f3edf3
HTTP/1.1" 500 956
http://olmedo-servicios-integrales.appspot.com/gae/watchdog
"AppEngine-Google; (+http://code.google.com/appengine)" "
olmedo-servicios-integrales.appspot.com" ms=550 cpu_ms=231
cpm_usd=2.5548e-7 loading_request=0
instance=00c61b117c327d751511c82e8ba427e6081789412cae8a6b2a33f690566e8b31
app_engine_release=1.9.38 trace_id=-

{

metadata: {…}
protoPayload: {…}
insertId: "2016-06-06|09:15:00.884913-07|10.106.162.86|1409825575"
log: "appengine.googleapis.com/request_log"
httpRequest: {…}
operation: {…} }

Seaching for OutOfMemory in the log and I don´t find anything.

Right now I just pulled all the information with Briefcase, and As far
as I can see I have more than what I had in my published tables.

My biggest concern is that I could loose submissions :S.

Any help is very appreciated.

Regards,

On Fri, May 6, 2016 at 6:22 PM, Mitch Sundt mitche...@gmail.com wrote:

This would indicate a data corruption issue.

See these instructions:
https://github.com/opendatakit/opendatakit/wiki/Aggregate-AppEngine-Troubleshooting#reparing-a-filled-in-form-submission

w.r.t. ODK Briefcase starting to pull data from the beginning every
time, if, after you pull data, you issue a push back up to the same server,
this will set tracking flags such that those records will be ignored and
skipped over the next time you pull.

ODK Briefcase stops at the first corrupted record because it is
fetching data in ascending marked-as-complete timestamp order. Until the
corrupted record is repaired or removed, it will not proceed to other
records.

Data corruption is more likely to occur if your server memory is
becoming exhausted or if you are submitting data over very-low-bandwidth
communications channels (e.g., satellite).

On Fri, May 6, 2016 at 5:52 AM, nuncaestardes...@gmail.com wrote:

El viernes, 6 de mayo de 2016, 12:30:39 (UTC+2), nuncaestardes...@gmail.com escribió:

El martes, 3 de mayo de 2016, 14:10:28 (UTC+2), nuncaestardes...@gmail.com escribió:

Hi guys,

I´m kind of new here, but I already have this problem:

On ODK Aggregate, when I want to export my data to .csv file, It´s
showing me an error (Failure - will retry later). it was working fine but I
guess the DDBB reach a certain volume that made it fail.

I have read that it is related to global max_allowed_packet which
has to be set to a higher value, with the following expression

set global max_allowed_packet = 1073741824;

I have my ODK aggregate deployed in a google app engine, so i tied
to type that in the cloud shell but nothing happened, I still have the same
problem, and as you can imaging it´s the worst moment to happend...

I have a excel macro that reads the csv file and download the media
files, renames and save everything in a file, so then i can open each
picture from a link in an excel spreadsheet.

Now I can only download the the CSV using ODK briefcase by:

1- Pulling all the data.

2- Exporting the CSV (I can only do it if I firstly pull the data)

The problem is that by doing this I have to download the media
files twice (in briefcase and in my macro), and the CSV file exported by
briefcase contains the local link in my computer to the media files, but
not the cloud link to the media files, which in my case it is super
convenient. I figure out that the link to the media is a concatenation of
the ID number and some other things, so I manage to bypass this problem,
but I had to change my whole procedure in the middle of the work, making it
much more manual and not allowing me to use the tools I developped to
automate all this work.

Anyone knows how can I in ODK aggregate deployed in google app
engine execute this procedure?

set global max_allowed_packet = 1073741824;

IF that really is the problem, which i hope it is.

Thank you very much in advance.

Regards,

Pablo

One more thing, everytime I pull information with briefcase it
downloads the whole project instead of just downloading the new
entries,that could be something easy to improve in future versions, my
excel macro that downloads everything from the csv does that it is muc more
convenient this way.

Regards,

Pablo

I am trying to pull all the information with briefcase, and looking at
the log I have this:

...

fetching instance 3048 ...

fetching instance 3049 ...

fetching instance 3050 ...

fetching instance 3051 ...

fetching instance 3052 ...

fetching instance 3053 ...

retrieving next chunk of instances from server...

FAILED.

I don´t know if it failed because there is nothing more to pull or
because actually there is something elses happening... right now I don´t
know how many entries I have so 3053? I don´t know... the filter in
aggregate does not work so i don´t know how could I do this...

--

--

Post: opend...@googlegroups.com

Unsubscribe: opendatakit...@googlegroups.com

Options: http://groups.google.com/group/opendatakit?hl=en

You received this message because you are subscribed to the Google
Groups "ODK Community" group.

To unsubscribe from this group and stop receiving emails from it, send
an email to opendatakit...@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--

Mitch Sundt
Software Engineer
University of Washington
mitche...@gmail.com

--

--

Post: opend...@googlegroups.com

Unsubscribe: opendatakit...@googlegroups.com

Options: http://groups.google.com/group/opendatakit?hl=en

You received this message because you are subscribed to a topic in the
Google Groups "ODK Community" group.

To unsubscribe from this topic, visit
https://groups.google.com/d/topic/opendatakit/hjMKcHkk1tc/unsubscribe.

To unsubscribe from this group and all its topics, send an email to
opendatakit...@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--

--

Post: opend...@googlegroups.com

Unsubscribe: opendatakit...@googlegroups.com

Options: http://groups.google.com/group/opendatakit?hl=en

You received this message because you are subscribed to the Google
Groups "ODK Community" group.

To unsubscribe from this group and stop receiving emails from it, send
an email to opendatakit...@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--

Mitch Sundt
Software Engineer
University of Washington
mitche...@gmail.com

--

Post: opendatakit@googlegroups.com
Unsubscribe: opendatakit+unsubscribe@googlegroups.com
Options: http://groups.google.com/group/opendatakit?hl=en

You received this message because you are subscribed to a topic in the
Google Groups "ODK Community" group.
To unsubscribe from this topic, visit
https://groups.google.com/d/topic/opendatakit/hjMKcHkk1tc/unsubscribe.
To unsubscribe from this group and all its topics, send an email to
opendatakit+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Pablo_Rodriguez · June 7, 2016, 3:32pm

Ok, I think I´m seeing some progress in the sense of figuring out, what´s
happening. I see where did you took the query you gave me, from Logs, App
Engine, Background, All versions:
[image: Inline image 1]
If I enlarge the first error I can see what was happening:
[image: Inline image 2]

And I followed your instructions. But there is a second error, and it is
different:
[image: Inline image 4]
where it says:

org.opendatakit.aggregate.task.CsvWorkerImpl failureRecovery: Exception
caught: org.opendatakit.common.datamodel.ODKEnumeratedElementException:
Attachment errors:
SELECT * FROM MAPFRE_JUNIO2016_G2_FOTO1_BN WHERE _TOP_LEVEL_AURI =
uuid:9e03e883-84bd-4457-a9cd-ae54746e1139 AND _PARENT_AURI =
uuid:9e03e883-84bd-4457-a9cd-ae54746e1139 is missing an attachment instance
OR has extra copies. for MapfreJunio2016
(BTW, in MAPFRE_JUNIO2016_G2_FOTO1_BN G2 stands for group1 since in this
form there are several fields displayed in the same screen, and FOTO1 means
"picture1" which is a required field... I don´t know if that is relevant)

Then I decided to query:

SELECT * FROM MAPFRE_JUNIO2016_G2_FOTO1_BN WHERE _TOP_LEVEL_AURI =
"uuid:9e03e883-84bd-4457-a9cd-ae54746e1139" AND _PARENT_AURI =
"uuid:9e03e883-84bd-4457-a9cd-ae54746e1139"

But in this case instead of finding 2 entities there is no match. Also I
can see that by searching this folder in the instances folder created by
ODK Briefcase, instead of having an XML file and a picture/s it is empty.

So I guess now I should find a way of deleting
uuid:9e03e883-84bd-4457-a9cd-ae54746e1139
in the database since it is a void submission, but I don´t know how to do
this. ¿how can I do this?

What about all the rest of errors that I am finding in APP Engine,Default
Service that I posted before? should I worry about them? (Because I do)

Why is this happening? How can I prevent this to happen again? could large
pictures be a problem? (right now the pulled process made by briefcase is
4,24GB which includes XML files and pictures. This is 65% of what I am
expecting)

Thank you very much in advance.

···

On Tue, Jun 7, 2016 at 10:09 AM, Pablo Rodríguez < nuncaestardesillegaspronto@gmail.com> wrote:

Ok So I see there are two errors constantly happening:
[image: Inline image 1]

one of them:

09:55:58.212GET500960 B5.9 sAppEngine-Google; (+
http://code.google.com/appengine)
/gae/uploadSubmissionsTask?fscUri=uuid%3A9672732c-1bc6-4dde-a463-d066b5f3edf3
0.1.0.2 - - [07/Jun/2016:09:55:58 +0200] "GET
/gae/uploadSubmissionsTask?fscUri=uuid%3A9672732c-1bc6-4dde-a463-d066b5f3edf3
HTTP/1.1" 500 960
http://olmedo-servicios-integrales.appspot.com/gae/watchdog
"AppEngine-Google; (+http://code.google.com/appengine)" "
olmedo-servicios-integrales.appspot.com" ms=5901 cpu_ms=460
cpm_usd=3.1940599999999997e-7 loading_request=0
instance=00c61b117c0e9e3d18748ca60cc00676c90a86e4344dfcefb7184ce68bcd8de904
app_engine_release=1.9.38 trace_id=-
{
metadata: {…}
protoPayload: {…}
insertId: "2016-06-07|00:56:08.408633-07|10.94.34.134|-1592936713"
log: "appengine.googleapis.com/request_log"
httpRequest: {…}
operation: {…}
}
09:56:01.221org.opendatakit.aggregate.task.gae.servlet.UploadSubmissionsTaskServlet
doGet: Beginning servlet processing
09:56:01.227org.opendatakit.aggregate.task.gae.servlet.UploadSubmissionsTaskServlet
doGet: Request is running on frontend
09:56:01.228org.opendatakit.aggregate.task.UploadSubmissionsWorkerImpl
uploadAllSubmissions: Beginning UPLOAD service:
uuid:8979e040-b496-49c1-b8de-f355d69d4c46 form MapfreJunio2016
09:56:01.228org.opendatakit.aggregate.form.FormFactory internalGetForms:
FormCache: using cached list of Forms
09:56:02.449org.opendatakit.aggregate.form.FormFactory internalGetForms:
FormCache: using cached list of Forms
09:56:02.478[s~olmedo-servicios-integrales/1.392851787586356529].:
Time left on lock: -73233127
09:56:02.478[s~olmedo-servicios-integrales/1.392851787586356529].:
Time left on lock: -13406599
09:56:02.478[s~olmedo-servicios-integrales/1.392851787586356529].:
Time left on lock: -52076838
09:56:02.478[s~olmedo-servicios-integrales/1.392851787586356529].:
Time left on lock: -57872843
09:56:02.478[s~olmedo-servicios-integrales/1.392851787586356529].:
Time left on lock: -78707824

And the other one

09:56:03.646GET2020 B1.4 sAppEngine-Google; (+
http://code.google.com/appengine)
/gae/uploadSubmissionsTask?fscUri=uuid%3A7ad28143-2a7a-4624-8807-215f05fd0102
0.1.0.2 - - [07/Jun/2016:09:56:03 +0200] "GET
/gae/uploadSubmissionsTask?fscUri=uuid%3A7ad28143-2a7a-4624-8807-215f05fd0102
HTTP/1.1" 202 -
http://olmedo-servicios-integrales.appspot.com/gae/watchdog
"AppEngine-Google; (+http://code.google.com/appengine)" "
olmedo-servicios-integrales.appspot.com" ms=1400 cpu_ms=120
cpm_usd=1.39586e-7 loading_request=0
instance=00c61b117c0e9e3d18748ca60cc00676c90a86e4344dfcefb7184ce68bcd8de904
app_engine_release=1.9.38 trace_id=-
{
metadata: {…}
protoPayload: {…}
insertId: "2016-06-07|00:56:08.408751-07|10.94.34.134|-1592936713"
log: "appengine.googleapis.com/request_log"
httpRequest: {…}
operation: {…}
}
09:56:03.652org.opendatakit.aggregate.task.gae.servlet.UploadSubmissionsTaskServlet
doGet: Beginning servlet processing
09:56:03.658org.opendatakit.aggregate.task.gae.servlet.UploadSubmissionsTaskServlet
doGet: Request is running on frontend
09:56:03.658org.opendatakit.aggregate.task.UploadSubmissionsWorkerImpl
uploadAllSubmissions: Beginning UPLOAD service:
uuid:28dbf50b-2fcf-4424-800f-9c90eb823a09 form MapfreJunio2016
09:56:03.658org.opendatakit.aggregate.form.FormFactory internalGetForms:
FormCache: using cached list of Forms
09:56:03.856org.opendatakit.aggregate.form.FormFactory internalGetForms:
FormCache: using cached list of Forms
09:56:03.881[s~olmedo-servicios-integrales/1.392851787586356529].:
Time left on lock: -14278749
09:56:03.881[s~olmedo-servicios-integrales/1.392851787586356529].:
Time left on lock: -17503047
09:56:03.881[s~olmedo-servicios-integrales/1.392851787586356529].:
Time left on lock: -34874322
09:56:03.881[s~olmedo-servicios-integrales/1.392851787586356529].:
Time left on lock: -50316469
09:56:03.881[s~olmedo-servicios-integrales/1.392851787586356529].:
Time left on lock: -18963331

So I try to run your query but changing the uuid to see if i can see two
entries like the first thing I did but it says no entities matched. I
checked and the publishing is still not working, but when I pull with
Briefcase I can see it is downloading more instances, so I guess the system
keeps receiving forms...
[image: Inline image 2]

On Tue, Jun 7, 2016 at 9:51 AM, Pablo Rodríguez < nuncaestardesillegaspronto@gmail.com> wrote:

I´ve done exactly what do told me, now I´m pulling with briefcase to see
if it works, but ends with FAILED
[image: Inline image 1]

The problem is that I´m struggling to understand what I am doing, so the
steps for Repairing a Filled-in Form Submission are not straight forward
for me. I´ll keep trying...

On Tue, Jun 7, 2016 at 9:22 AM, nuncaestardesillegaspronto@gmail.com wrote:

I´m going to try Mitch, for me this is dancing on thin ice... let´s
see...!

Thanks!

El martes, 7 de junio de 2016, 0:55:47 (UTC+2), Mitch Sundt escribió:

Have you worked through the database repair steps described here:

https://github.com/opendatakit/opendatakit/wiki/Aggregate-AppEngine-Troubleshooting#repairing-a-filled-in-form-submission

I.e.,

org.opendatakit.aggregate.task.gae.servlet.UploadSubmissionsTaskServlet
doGet: org.opendatakit.aggregate.exception.ODKExternalServiceException:
org.opendatakit.common.datamodel.ODKEnumeratedElementException: SELECT *
FROM MAPFRE_JUNIO2016_G3_REPORTE WHERE _TOP_LEVEL_AURI =
uuid:9e03e883-84bd-4457-a9cd-ae54746e1139 AND _PARENT_AURI =
uuid:9e03e883-84bd-4457-a9cd-ae54746e1139 multiple-choice valueSet is
missing an entry OR has an extra copy of one

and then, on the datastore tab,

entering in the query box:

SELECT * FROM opendatakit.MAPFRE_JUNIO2016_G3_REPORTE WHERE
_TOP_LEVEL_AURI = "uuid:9e03e883-84bd-4457-a9cd-ae54746e1139" AND
_PARENT_AURI = "uuid:9e03e883-84bd-4457-a9cd-ae54746e1139"

you'll find 2 entries with matching _ORDINAL_NUMBER values (1).

Per the instructions (step 6 & 7), delete the older one of these.

On Mon, Jun 6, 2016 at 9:39 AM, Pablo Rodríguez < nuncaestardes...@gmail.com> wrote:

Well Now in another project I´m having the very same issue. wort
timing ever. In the middle of a big running project :S

There are not incomplete submissions according to ODK.

I was publishing a google spreadsheet and a fusion table (Running in
google app engine with billing enabled)

I´m looking at the log and I´m having an average of 4 to 7 error per
minute, basically like this:

18:14:55.688GET500956 B550 msAppEngine-Google; (+
http://code.google.com/appengine)/gae/uploadSubmissionsTask?fscUri=uuid%3A9672732c-1bc6-4dde-a463-d066b5f3edf3

0.1.0.2 - - [06/Jun/2016:18:14:55 +0200] "GET
/gae/uploadSubmissionsTask?fscUri=uuid%3A9672732c-1bc6-4dde-a463-d066b5f3edf3
HTTP/1.1" 500 956
http://olmedo-servicios-integrales.appspot.com/gae/watchdog
"AppEngine-Google; (+http://code.google.com/appengine)" "
olmedo-servicios-integrales.appspot.com" ms=550 cpu_ms=231
cpm_usd=2.5548e-7 loading_request=0
instance=00c61b117c327d751511c82e8ba427e6081789412cae8a6b2a33f690566e8b31
app_engine_release=1.9.38 trace_id=-

{

metadata: {…}
protoPayload: {…}
insertId: "2016-06-06|09:15:00.884913-07|10.106.162.86|1409825575"
log: "appengine.googleapis.com/request_log"
httpRequest: {…}
operation: {…} }

Seaching for OutOfMemory in the log and I don´t find anything.

Right now I just pulled all the information with Briefcase, and As far
as I can see I have more than what I had in my published tables.

My biggest concern is that I could loose submissions :S.

Any help is very appreciated.

Regards,

On Fri, May 6, 2016 at 6:22 PM, Mitch Sundt mitche...@gmail.com wrote:

This would indicate a data corruption issue.

See these instructions:
https://github.com/opendatakit/opendatakit/wiki/Aggregate-AppEngine-Troubleshooting#reparing-a-filled-in-form-submission

w.r.t. ODK Briefcase starting to pull data from the beginning every
time, if, after you pull data, you issue a push back up to the same server,
this will set tracking flags such that those records will be ignored and
skipped over the next time you pull.

ODK Briefcase stops at the first corrupted record because it is
fetching data in ascending marked-as-complete timestamp order. Until the
corrupted record is repaired or removed, it will not proceed to other
records.

Data corruption is more likely to occur if your server memory is
becoming exhausted or if you are submitting data over very-low-bandwidth
communications channels (e.g., satellite).

On Fri, May 6, 2016 at 5:52 AM, nuncaestardes...@gmail.com wrote:

El viernes, 6 de mayo de 2016, 12:30:39 (UTC+2), nuncaestardes...@gmail.com escribió:

El martes, 3 de mayo de 2016, 14:10:28 (UTC+2), nuncaestardes...@gmail.com escribió:

Hi guys,

I´m kind of new here, but I already have this problem:

On ODK Aggregate, when I want to export my data to .csv file, It´s
showing me an error (Failure - will retry later). it was working fine but I
guess the DDBB reach a certain volume that made it fail.

I have read that it is related to global max_allowed_packet which
has to be set to a higher value, with the following expression

set global max_allowed_packet = 1073741824;

I have my ODK aggregate deployed in a google app engine, so i tied
to type that in the cloud shell but nothing happened, I still have the same
problem, and as you can imaging it´s the worst moment to happend...

I have a excel macro that reads the csv file and download the
media files, renames and save everything in a file, so then i can open each
picture from a link in an excel spreadsheet.

Now I can only download the the CSV using ODK briefcase by:

1- Pulling all the data.

2- Exporting the CSV (I can only do it if I firstly pull the data)

The problem is that by doing this I have to download the media
files twice (in briefcase and in my macro), and the CSV file exported by
briefcase contains the local link in my computer to the media files, but
not the cloud link to the media files, which in my case it is super
convenient. I figure out that the link to the media is a concatenation of
the ID number and some other things, so I manage to bypass this problem,
but I had to change my whole procedure in the middle of the work, making it
much more manual and not allowing me to use the tools I developped to
automate all this work.

Anyone knows how can I in ODK aggregate deployed in google app
engine execute this procedure?

set global max_allowed_packet = 1073741824;

IF that really is the problem, which i hope it is.

Thank you very much in advance.

Regards,

Pablo

One more thing, everytime I pull information with briefcase it
downloads the whole project instead of just downloading the new
entries,that could be something easy to improve in future versions, my
excel macro that downloads everything from the csv does that it is muc more
convenient this way.

Regards,

Pablo

I am trying to pull all the information with briefcase, and looking at
the log I have this:

...

fetching instance 3048 ...

fetching instance 3049 ...

fetching instance 3050 ...

fetching instance 3051 ...

fetching instance 3052 ...

fetching instance 3053 ...

retrieving next chunk of instances from server...

FAILED.

I don´t know if it failed because there is nothing more to pull or
because actually there is something elses happening... right now I don´t
know how many entries I have so 3053? I don´t know... the filter in
aggregate does not work so i don´t know how could I do this...

--

--

Post: opend...@googlegroups.com

Unsubscribe: opendatakit...@googlegroups.com

Options: http://groups.google.com/group/opendatakit?hl=en

You received this message because you are subscribed to the Google
Groups "ODK Community" group.

To unsubscribe from this group and stop receiving emails from it, send
an email to opendatakit...@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--

Mitch Sundt
Software Engineer
University of Washington
mitche...@gmail.com

--

--

Post: opend...@googlegroups.com

Unsubscribe: opendatakit...@googlegroups.com

Options: http://groups.google.com/group/opendatakit?hl=en

You received this message because you are subscribed to a topic in the
Google Groups "ODK Community" group.

To unsubscribe from this topic, visit
https://groups.google.com/d/topic/opendatakit/hjMKcHkk1tc/unsubscribe.

To unsubscribe from this group and all its topics, send an email to
opendatakit...@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--

--

Post: opend...@googlegroups.com

Unsubscribe: opendatakit...@googlegroups.com

Options: http://groups.google.com/group/opendatakit?hl=en

You received this message because you are subscribed to the Google
Groups "ODK Community" group.

To unsubscribe from this group and stop receiving emails from it, send
an email to opendatakit...@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--

Mitch Sundt
Software Engineer
University of Washington
mitche...@gmail.com

--

Post: opendatakit@googlegroups.com
Unsubscribe: opendatakit+unsubscribe@googlegroups.com
Options: http://groups.google.com/group/opendatakit?hl=en

You received this message because you are subscribed to a topic in the
Google Groups "ODK Community" group.
To unsubscribe from this topic, visit
https://groups.google.com/d/topic/opendatakit/hjMKcHkk1tc/unsubscribe.
To unsubscribe from this group and all its topics, send an email to
opendatakit+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Pablo_Rodriguez · June 7, 2016, 5:05pm

One last thing and hoping I can fix the main problem, since this is
secondary: I tried to publish another spreadsheet (didnt´t work) and
another fusion table (just existing submissions, 1420, when I had already
published 1476 in the original) again, but at the end I decided to delete
them. The two original spreadsheet and fusion table are both "BOTH Upload
Existing & Stream New Submission Data" but now the status shows "PAUSED".
How can I make them active again?

···

On Tue, Jun 7, 2016 at 5:32 PM, Pablo Rodríguez < nuncaestardesillegaspronto@gmail.com> wrote:

Ok, I think I´m seeing some progress in the sense of figuring out, what´s
happening. I see where did you took the query you gave me, from Logs, App
Engine, Background, All versions:
[image: Inline image 1]
If I enlarge the first error I can see what was happening:
[image: Inline image 2]

And I followed your instructions. But there is a second error, and it is
different:
[image: Inline image 4]
where it says:

org.opendatakit.aggregate.task.CsvWorkerImpl failureRecovery: Exception
caught: org.opendatakit.common.datamodel.ODKEnumeratedElementException:
Attachment errors:
SELECT * FROM MAPFRE_JUNIO2016_G2_FOTO1_BN WHERE _TOP_LEVEL_AURI =
uuid:9e03e883-84bd-4457-a9cd-ae54746e1139 AND _PARENT_AURI =
uuid:9e03e883-84bd-4457-a9cd-ae54746e1139 is missing an attachment instance
OR has extra copies. for MapfreJunio2016
(BTW, in MAPFRE_JUNIO2016_G2_FOTO1_BN G2 stands for group1 since in this
form there are several fields displayed in the same screen, and FOTO1 means
"picture1" which is a required field... I don´t know if that is relevant)

Then I decided to query:

SELECT * FROM MAPFRE_JUNIO2016_G2_FOTO1_BN WHERE _TOP_LEVEL_AURI =
"uuid:9e03e883-84bd-4457-a9cd-ae54746e1139" AND _PARENT_AURI =
"uuid:9e03e883-84bd-4457-a9cd-ae54746e1139"

But in this case instead of finding 2 entities there is no match. Also I
can see that by searching this folder in the instances folder created by
ODK Briefcase, instead of having an XML file and a picture/s it is empty.

So I guess now I should find a way of deleting uuid:9e03e883-84bd-4457-a9cd-ae54746e1139
in the database since it is a void submission, but I don´t know how to do
this. ¿how can I do this?

What about all the rest of errors that I am finding in APP Engine,Default
Service that I posted before? should I worry about them? (Because I do)

Why is this happening? How can I prevent this to happen again? could large
pictures be a problem? (right now the pulled process made by briefcase is
4,24GB which includes XML files and pictures. This is 65% of what I am
expecting)

Thank you very much in advance.

On Tue, Jun 7, 2016 at 10:09 AM, Pablo Rodríguez < nuncaestardesillegaspronto@gmail.com> wrote:

Ok So I see there are two errors constantly happening:
[image: Inline image 1]

one of them:

09:55:58.212GET500960 B5.9 sAppEngine-Google; (+
http://code.google.com/appengine)
/gae/uploadSubmissionsTask?fscUri=uuid%3A9672732c-1bc6-4dde-a463-d066b5f3edf3
0.1.0.2 - - [07/Jun/2016:09:55:58 +0200] "GET
/gae/uploadSubmissionsTask?fscUri=uuid%3A9672732c-1bc6-4dde-a463-d066b5f3edf3
HTTP/1.1" 500 960
http://olmedo-servicios-integrales.appspot.com/gae/watchdog
"AppEngine-Google; (+http://code.google.com/appengine)" "
olmedo-servicios-integrales.appspot.com" ms=5901 cpu_ms=460
cpm_usd=3.1940599999999997e-7 loading_request=0
instance=00c61b117c0e9e3d18748ca60cc00676c90a86e4344dfcefb7184ce68bcd8de904
app_engine_release=1.9.38 trace_id=-
{
metadata: {…}
protoPayload: {…}
insertId: "2016-06-07|00:56:08.408633-07|10.94.34.134|-1592936713"
log: "appengine.googleapis.com/request_log"
httpRequest: {…}
operation: {…}
}
09:56:01.221org.opendatakit.aggregate.task.gae.servlet.UploadSubmissionsTaskServlet
doGet: Beginning servlet processing
09:56:01.227org.opendatakit.aggregate.task.gae.servlet.UploadSubmissionsTaskServlet
doGet: Request is running on frontend
09:56:01.228org.opendatakit.aggregate.task.UploadSubmissionsWorkerImpl
uploadAllSubmissions: Beginning UPLOAD service:
uuid:8979e040-b496-49c1-b8de-f355d69d4c46 form MapfreJunio2016
09:56:01.228org.opendatakit.aggregate.form.FormFactory internalGetForms:
FormCache: using cached list of Forms
09:56:02.449org.opendatakit.aggregate.form.FormFactory internalGetForms:
FormCache: using cached list of Forms
09:56:02.478[s~olmedo-servicios-integrales/1.392851787586356529].:
Time left on lock: -73233127
09:56:02.478[s~olmedo-servicios-integrales/1.392851787586356529].:
Time left on lock: -13406599
09:56:02.478[s~olmedo-servicios-integrales/1.392851787586356529].:
Time left on lock: -52076838
09:56:02.478[s~olmedo-servicios-integrales/1.392851787586356529].:
Time left on lock: -57872843
09:56:02.478[s~olmedo-servicios-integrales/1.392851787586356529].:
Time left on lock: -78707824

And the other one

09:56:03.646GET2020 B1.4 sAppEngine-Google; (+
http://code.google.com/appengine)
/gae/uploadSubmissionsTask?fscUri=uuid%3A7ad28143-2a7a-4624-8807-215f05fd0102
0.1.0.2 - - [07/Jun/2016:09:56:03 +0200] "GET
/gae/uploadSubmissionsTask?fscUri=uuid%3A7ad28143-2a7a-4624-8807-215f05fd0102
HTTP/1.1" 202 -
http://olmedo-servicios-integrales.appspot.com/gae/watchdog
"AppEngine-Google; (+http://code.google.com/appengine)" "
olmedo-servicios-integrales.appspot.com" ms=1400 cpu_ms=120
cpm_usd=1.39586e-7 loading_request=0
instance=00c61b117c0e9e3d18748ca60cc00676c90a86e4344dfcefb7184ce68bcd8de904
app_engine_release=1.9.38 trace_id=-
{
metadata: {…}
protoPayload: {…}
insertId: "2016-06-07|00:56:08.408751-07|10.94.34.134|-1592936713"
log: "appengine.googleapis.com/request_log"
httpRequest: {…}
operation: {…}
}
09:56:03.652org.opendatakit.aggregate.task.gae.servlet.UploadSubmissionsTaskServlet
doGet: Beginning servlet processing
09:56:03.658org.opendatakit.aggregate.task.gae.servlet.UploadSubmissionsTaskServlet
doGet: Request is running on frontend
09:56:03.658org.opendatakit.aggregate.task.UploadSubmissionsWorkerImpl
uploadAllSubmissions: Beginning UPLOAD service:
uuid:28dbf50b-2fcf-4424-800f-9c90eb823a09 form MapfreJunio2016
09:56:03.658org.opendatakit.aggregate.form.FormFactory internalGetForms:
FormCache: using cached list of Forms
09:56:03.856org.opendatakit.aggregate.form.FormFactory internalGetForms:
FormCache: using cached list of Forms
09:56:03.881[s~olmedo-servicios-integrales/1.392851787586356529].:
Time left on lock: -14278749
09:56:03.881[s~olmedo-servicios-integrales/1.392851787586356529].:
Time left on lock: -17503047
09:56:03.881[s~olmedo-servicios-integrales/1.392851787586356529].:
Time left on lock: -34874322
09:56:03.881[s~olmedo-servicios-integrales/1.392851787586356529].:
Time left on lock: -50316469
09:56:03.881[s~olmedo-servicios-integrales/1.392851787586356529].:
Time left on lock: -18963331

So I try to run your query but changing the uuid to see if i can see two
entries like the first thing I did but it says no entities matched. I
checked and the publishing is still not working, but when I pull with
Briefcase I can see it is downloading more instances, so I guess the system
keeps receiving forms...
[image: Inline image 2]

On Tue, Jun 7, 2016 at 9:51 AM, Pablo Rodríguez < nuncaestardesillegaspronto@gmail.com> wrote:

I´ve done exactly what do told me, now I´m pulling with briefcase to see
if it works, but ends with FAILED
[image: Inline image 1]

The problem is that I´m struggling to understand what I am doing, so the
steps for Repairing a Filled-in Form Submission are not straight forward
for me. I´ll keep trying...

On Tue, Jun 7, 2016 at 9:22 AM, nuncaestardesillegaspronto@gmail.com wrote:

I´m going to try Mitch, for me this is dancing on thin ice... let´s
see...!

Thanks!

El martes, 7 de junio de 2016, 0:55:47 (UTC+2), Mitch Sundt escribió:

Have you worked through the database repair steps described here:

https://github.com/opendatakit/opendatakit/wiki/Aggregate-AppEngine-Troubleshooting#repairing-a-filled-in-form-submission

I.e.,

org.opendatakit.aggregate.task.gae.servlet.UploadSubmissionsTaskServlet
doGet: org.opendatakit.aggregate.exception.ODKExternalServiceException:
org.opendatakit.common.datamodel.ODKEnumeratedElementException: SELECT *
FROM MAPFRE_JUNIO2016_G3_REPORTE WHERE _TOP_LEVEL_AURI =
uuid:9e03e883-84bd-4457-a9cd-ae54746e1139 AND _PARENT_AURI =
uuid:9e03e883-84bd-4457-a9cd-ae54746e1139 multiple-choice valueSet is
missing an entry OR has an extra copy of one

and then, on the datastore tab,

entering in the query box:

SELECT * FROM opendatakit.MAPFRE_JUNIO2016_G3_REPORTE WHERE
_TOP_LEVEL_AURI = "uuid:9e03e883-84bd-4457-a9cd-ae54746e1139" AND
_PARENT_AURI = "uuid:9e03e883-84bd-4457-a9cd-ae54746e1139"

you'll find 2 entries with matching _ORDINAL_NUMBER values (1).

Per the instructions (step 6 & 7), delete the older one of these.

On Mon, Jun 6, 2016 at 9:39 AM, Pablo Rodríguez < nuncaestardes...@gmail.com> wrote:

Well Now in another project I´m having the very same issue. wort
timing ever. In the middle of a big running project :S

There are not incomplete submissions according to ODK.

I was publishing a google spreadsheet and a fusion table (Running in
google app engine with billing enabled)

I´m looking at the log and I´m having an average of 4 to 7 error per
minute, basically like this:

18:14:55.688GET500956 B550 msAppEngine-Google; (+
http://code.google.com/appengine)/gae/uploadSubmissionsTask?fscUri=uuid%3A9672732c-1bc6-4dde-a463-d066b5f3edf3

0.1.0.2 - - [06/Jun/2016:18:14:55 +0200] "GET
/gae/uploadSubmissionsTask?fscUri=uuid%3A9672732c-1bc6-4dde-a463-d066b5f3edf3
HTTP/1.1" 500 956
http://olmedo-servicios-integrales.appspot.com/gae/watchdog
"AppEngine-Google; (+http://code.google.com/appengine)" "
olmedo-servicios-integrales.appspot.com" ms=550 cpu_ms=231
cpm_usd=2.5548e-7 loading_request=0
instance=00c61b117c327d751511c82e8ba427e6081789412cae8a6b2a33f690566e8b31
app_engine_release=1.9.38 trace_id=-

{

metadata: {…}
protoPayload: {…}
insertId: "2016-06-06|09:15:00.884913-07|10.106.162.86|1409825575"
log: "appengine.googleapis.com/request_log"
httpRequest: {…}
operation: {…} }

Seaching for OutOfMemory in the log and I don´t find anything.

Right now I just pulled all the information with Briefcase, and As
far as I can see I have more than what I had in my published tables.

My biggest concern is that I could loose submissions :S.

Any help is very appreciated.

Regards,

On Fri, May 6, 2016 at 6:22 PM, Mitch Sundt mitche...@gmail.com wrote:

This would indicate a data corruption issue.

See these instructions:
https://github.com/opendatakit/opendatakit/wiki/Aggregate-AppEngine-Troubleshooting#reparing-a-filled-in-form-submission

w.r.t. ODK Briefcase starting to pull data from the beginning every
time, if, after you pull data, you issue a push back up to the same server,
this will set tracking flags such that those records will be ignored and
skipped over the next time you pull.

ODK Briefcase stops at the first corrupted record because it is
fetching data in ascending marked-as-complete timestamp order. Until the
corrupted record is repaired or removed, it will not proceed to other
records.

Data corruption is more likely to occur if your server memory is
becoming exhausted or if you are submitting data over very-low-bandwidth
communications channels (e.g., satellite).

On Fri, May 6, 2016 at 5:52 AM, nuncaestardes...@gmail.com wrote:

El viernes, 6 de mayo de 2016, 12:30:39 (UTC+2), nuncaestardes...@gmail.com escribió:

El martes, 3 de mayo de 2016, 14:10:28 (UTC+2), nuncaestardes...@gmail.com escribió:

Hi guys,

I´m kind of new here, but I already have this problem:

On ODK Aggregate, when I want to export my data to .csv file,
It´s showing me an error (Failure - will retry later). it was working fine
but I guess the DDBB reach a certain volume that made it fail.

I have read that it is related to global max_allowed_packet which
has to be set to a higher value, with the following expression

set global max_allowed_packet = 1073741824;

I have my ODK aggregate deployed in a google app engine, so i
tied to type that in the cloud shell but nothing happened, I still have the
same problem, and as you can imaging it´s the worst moment to happend...

I have a excel macro that reads the csv file and download the
media files, renames and save everything in a file, so then i can open each
picture from a link in an excel spreadsheet.

Now I can only download the the CSV using ODK briefcase by:

1- Pulling all the data.

2- Exporting the CSV (I can only do it if I firstly pull the data)

The problem is that by doing this I have to download the media
files twice (in briefcase and in my macro), and the CSV file exported by
briefcase contains the local link in my computer to the media files, but
not the cloud link to the media files, which in my case it is super
convenient. I figure out that the link to the media is a concatenation of
the ID number and some other things, so I manage to bypass this problem,
but I had to change my whole procedure in the middle of the work, making it
much more manual and not allowing me to use the tools I developped to
automate all this work.

Anyone knows how can I in ODK aggregate deployed in google app
engine execute this procedure?

set global max_allowed_packet = 1073741824;

IF that really is the problem, which i hope it is.

Thank you very much in advance.

Regards,

Pablo

One more thing, everytime I pull information with briefcase it
downloads the whole project instead of just downloading the new
entries,that could be something easy to improve in future versions, my
excel macro that downloads everything from the csv does that it is muc more
convenient this way.

Regards,

Pablo

I am trying to pull all the information with briefcase, and looking
at the log I have this:

...

fetching instance 3048 ...

fetching instance 3049 ...

fetching instance 3050 ...

fetching instance 3051 ...

fetching instance 3052 ...

fetching instance 3053 ...

retrieving next chunk of instances from server...

FAILED.

I don´t know if it failed because there is nothing more to pull or
because actually there is something elses happening... right now I don´t
know how many entries I have so 3053? I don´t know... the filter in
aggregate does not work so i don´t know how could I do this...

--

--

Post: opend...@googlegroups.com

Unsubscribe: opendatakit...@googlegroups.com

Options: http://groups.google.com/group/opendatakit?hl=en

You received this message because you are subscribed to the Google
Groups "ODK Community" group.

To unsubscribe from this group and stop receiving emails from it,
send an email to opendatakit...@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--

Mitch Sundt
Software Engineer
University of Washington
mitche...@gmail.com

--

--

Post: opend...@googlegroups.com

Unsubscribe: opendatakit...@googlegroups.com

Options: http://groups.google.com/group/opendatakit?hl=en

You received this message because you are subscribed to a topic in
the Google Groups "ODK Community" group.

To unsubscribe from this topic, visit
https://groups.google.com/d/topic/opendatakit/hjMKcHkk1tc/unsubscribe.

To unsubscribe from this group and all its topics, send an email to
opendatakit...@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--

--

Post: opend...@googlegroups.com

Unsubscribe: opendatakit...@googlegroups.com

Options: http://groups.google.com/group/opendatakit?hl=en

You received this message because you are subscribed to the Google
Groups "ODK Community" group.

To unsubscribe from this group and stop receiving emails from it,
send an email to opendatakit...@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--

Mitch Sundt
Software Engineer
University of Washington
mitche...@gmail.com

--

Post: opendatakit@googlegroups.com
Unsubscribe: opendatakit+unsubscribe@googlegroups.com
Options: http://groups.google.com/group/opendatakit?hl=en

You received this message because you are subscribed to a topic in the
Google Groups "ODK Community" group.
To unsubscribe from this topic, visit
https://groups.google.com/d/topic/opendatakit/hjMKcHkk1tc/unsubscribe.
To unsubscribe from this group and all its topics, send an email to
opendatakit+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Mitch_S · June 7, 2016, 9:26pm

You are very close. Note that in step (5), the instructions say:

Now, for each data table name and XXXXX value in step (2), execute a
query of the form:

SELECT * FROM opendatakit.YOUR_TABLE_NAME WHERE _PARENT_AURI = "XXXXXX"

Take special note of the backquote around the opendatakit.-qualified table
name, and the double-quotes around the XXXXXX value. If the original query
specified _DOM_AURI instead of_PARENT_AURI, use that. If this query does
not return a value, you may also need to add the AND _TOP_LEVEL_AURI =
"YYYYY" qualifier (but that should not be necessary), if it was present in
the original error message.

In the queries you tried, you missed this part of the instructions
(replace MAPFRE_JUNIO2016_G2_FOTO1_BN
in your select statements with *opendatakit.*MAPFRE_JUNIO2016_G2_FOTO1_BN
)

These instructions are saying to select and show to you the entries in the
data tables that correspond to a specific submission (the one with
instanceID uuid:9e03e883-84bd-4457-a9cd-ae54746e1139 ).

···

======================== If the publisher is paused, I believe the "PAUSED" is actually a button, and you can click on that to get it to resume.

That, of course, will only be successful if you fix the data corruption.

On Tue, Jun 7, 2016 at 10:05 AM, Pablo Rodríguez < nuncaestardesillegaspronto@gmail.com> wrote:

One last thing and hoping I can fix the main problem, since this is
secondary: I tried to publish another spreadsheet (didnt´t work) and
another fusion table (just existing submissions, 1420, when I had already
published 1476 in the original) again, but at the end I decided to delete
them. The two original spreadsheet and fusion table are both "BOTH
Upload Existing & Stream New Submission Data" but now the status shows
"PAUSED". How can I make them active again?

On Tue, Jun 7, 2016 at 5:32 PM, Pablo Rodríguez < nuncaestardesillegaspronto@gmail.com> wrote:

Ok, I think I´m seeing some progress in the sense of figuring out, what´s
happening. I see where did you took the query you gave me, from Logs, App
Engine, Background, All versions:
[image: Inline image 1]
If I enlarge the first error I can see what was happening:
[image: Inline image 2]

And I followed your instructions. But there is a second error, and it is
different:
[image: Inline image 4]
where it says:

org.opendatakit.aggregate.task.CsvWorkerImpl failureRecovery: Exception
caught: org.opendatakit.common.datamodel.ODKEnumeratedElementException:
Attachment errors:
SELECT * FROM MAPFRE_JUNIO2016_G2_FOTO1_BN WHERE _TOP_LEVEL_AURI =
uuid:9e03e883-84bd-4457-a9cd-ae54746e1139 AND _PARENT_AURI =
uuid:9e03e883-84bd-4457-a9cd-ae54746e1139 is missing an attachment instance
OR has extra copies. for MapfreJunio2016
(BTW, in MAPFRE_JUNIO2016_G2_FOTO1_BN G2 stands for group1 since in this
form there are several fields displayed in the same screen, and FOTO1 means
"picture1" which is a required field... I don´t know if that is relevant)

Then I decided to query:

SELECT * FROM MAPFRE_JUNIO2016_G2_FOTO1_BN WHERE _TOP_LEVEL_AURI =
"uuid:9e03e883-84bd-4457-a9cd-ae54746e1139" AND _PARENT_AURI =
"uuid:9e03e883-84bd-4457-a9cd-ae54746e1139"

But in this case instead of finding 2 entities there is no match. Also I
can see that by searching this folder in the instances folder created by
ODK Briefcase, instead of having an XML file and a picture/s it is empty.

So I guess now I should find a way of deleting uuid:9e03e883-84bd-4457-a9cd-ae54746e1139
in the database since it is a void submission, but I don´t know how to do
this. ¿how can I do this?

What about all the rest of errors that I am finding in APP Engine,Default
Service that I posted before? should I worry about them? (Because I do)

Why is this happening? How can I prevent this to happen again? could
large pictures be a problem? (right now the pulled process made by
briefcase is 4,24GB which includes XML files and pictures. This is 65% of
what I am expecting)

Thank you very much in advance.

On Tue, Jun 7, 2016 at 10:09 AM, Pablo Rodríguez < nuncaestardesillegaspronto@gmail.com> wrote:

Ok So I see there are two errors constantly happening:
[image: Inline image 1]

one of them:

09:55:58.212GET500960 B5.9 sAppEngine-Google; (+
http://code.google.com/appengine)
/gae/uploadSubmissionsTask?fscUri=uuid%3A9672732c-1bc6-4dde-a463-d066b5f3edf3
0.1.0.2 - - [07/Jun/2016:09:55:58 +0200] "GET
/gae/uploadSubmissionsTask?fscUri=uuid%3A9672732c-1bc6-4dde-a463-d066b5f3edf3
HTTP/1.1" 500 960
http://olmedo-servicios-integrales.appspot.com/gae/watchdog
"AppEngine-Google; (+http://code.google.com/appengine)" "
olmedo-servicios-integrales.appspot.com" ms=5901 cpu_ms=460
cpm_usd=3.1940599999999997e-7 loading_request=0
instance=00c61b117c0e9e3d18748ca60cc00676c90a86e4344dfcefb7184ce68bcd8de904
app_engine_release=1.9.38 trace_id=-
{
metadata: {…}
protoPayload: {…}
insertId: "2016-06-07|00:56:08.408633-07|10.94.34.134|-1592936713"
log: "appengine.googleapis.com/request_log"
httpRequest: {…}
operation: {…}
}
09:56:01.221org.opendatakit.aggregate.task.gae.servlet.UploadSubmissionsTaskServlet
doGet: Beginning servlet processing
09:56:01.227org.opendatakit.aggregate.task.gae.servlet.UploadSubmissionsTaskServlet
doGet: Request is running on frontend
09:56:01.228org.opendatakit.aggregate.task.UploadSubmissionsWorkerImpl
uploadAllSubmissions: Beginning UPLOAD service:
uuid:8979e040-b496-49c1-b8de-f355d69d4c46 form MapfreJunio2016
09:56:01.228org.opendatakit.aggregate.form.FormFactory internalGetForms:
FormCache: using cached list of Forms
09:56:02.449org.opendatakit.aggregate.form.FormFactory internalGetForms:
FormCache: using cached list of Forms
09:56:02.478[s~olmedo-servicios-integrales/1.392851787586356529].:
Time left on lock: -73233127
09:56:02.478[s~olmedo-servicios-integrales/1.392851787586356529].:
Time left on lock: -13406599
09:56:02.478[s~olmedo-servicios-integrales/1.392851787586356529].:
Time left on lock: -52076838
09:56:02.478[s~olmedo-servicios-integrales/1.392851787586356529].:
Time left on lock: -57872843
09:56:02.478[s~olmedo-servicios-integrales/1.392851787586356529].:
Time left on lock: -78707824

And the other one

09:56:03.646GET2020 B1.4 sAppEngine-Google; (+
http://code.google.com/appengine)
/gae/uploadSubmissionsTask?fscUri=uuid%3A7ad28143-2a7a-4624-8807-215f05fd0102
0.1.0.2 - - [07/Jun/2016:09:56:03 +0200] "GET
/gae/uploadSubmissionsTask?fscUri=uuid%3A7ad28143-2a7a-4624-8807-215f05fd0102
HTTP/1.1" 202 -
http://olmedo-servicios-integrales.appspot.com/gae/watchdog
"AppEngine-Google; (+http://code.google.com/appengine)" "
olmedo-servicios-integrales.appspot.com" ms=1400 cpu_ms=120
cpm_usd=1.39586e-7 loading_request=0
instance=00c61b117c0e9e3d18748ca60cc00676c90a86e4344dfcefb7184ce68bcd8de904
app_engine_release=1.9.38 trace_id=-
{
metadata: {…}
protoPayload: {…}
insertId: "2016-06-07|00:56:08.408751-07|10.94.34.134|-1592936713"
log: "appengine.googleapis.com/request_log"
httpRequest: {…}
operation: {…}
}
09:56:03.652org.opendatakit.aggregate.task.gae.servlet.UploadSubmissionsTaskServlet
doGet: Beginning servlet processing
09:56:03.658org.opendatakit.aggregate.task.gae.servlet.UploadSubmissionsTaskServlet
doGet: Request is running on frontend
09:56:03.658org.opendatakit.aggregate.task.UploadSubmissionsWorkerImpl
uploadAllSubmissions: Beginning UPLOAD service:
uuid:28dbf50b-2fcf-4424-800f-9c90eb823a09 form MapfreJunio2016
09:56:03.658org.opendatakit.aggregate.form.FormFactory internalGetForms:
FormCache: using cached list of Forms
09:56:03.856org.opendatakit.aggregate.form.FormFactory internalGetForms:
FormCache: using cached list of Forms
09:56:03.881[s~olmedo-servicios-integrales/1.392851787586356529].:
Time left on lock: -14278749
09:56:03.881[s~olmedo-servicios-integrales/1.392851787586356529].:
Time left on lock: -17503047
09:56:03.881[s~olmedo-servicios-integrales/1.392851787586356529].:
Time left on lock: -34874322
09:56:03.881[s~olmedo-servicios-integrales/1.392851787586356529].:
Time left on lock: -50316469
09:56:03.881[s~olmedo-servicios-integrales/1.392851787586356529].:
Time left on lock: -18963331

So I try to run your query but changing the uuid to see if i can see two
entries like the first thing I did but it says no entities matched. I
checked and the publishing is still not working, but when I pull with
Briefcase I can see it is downloading more instances, so I guess the system
keeps receiving forms...
[image: Inline image 2]

On Tue, Jun 7, 2016 at 9:51 AM, Pablo Rodríguez < nuncaestardesillegaspronto@gmail.com> wrote:

I´ve done exactly what do told me, now I´m pulling with briefcase to
see if it works, but ends with FAILED
[image: Inline image 1]

The problem is that I´m struggling to understand what I am doing, so
the steps for Repairing a Filled-in Form Submission are not straight
forward for me. I´ll keep trying...

On Tue, Jun 7, 2016 at 9:22 AM, nuncaestardesillegaspronto@gmail.com wrote:

I´m going to try Mitch, for me this is dancing on thin ice... let´s
see...!

Thanks!

El martes, 7 de junio de 2016, 0:55:47 (UTC+2), Mitch Sundt escribió:

Have you worked through the database repair steps described here:

https://github.com/opendatakit/opendatakit/wiki/Aggregate-AppEngine-Troubleshooting#repairing-a-filled-in-form-submission

I.e.,

org.opendatakit.aggregate.task.gae.servlet.UploadSubmissionsTaskServlet
doGet: org.opendatakit.aggregate.exception.ODKExternalServiceException:
org.opendatakit.common.datamodel.ODKEnumeratedElementException: SELECT *
FROM MAPFRE_JUNIO2016_G3_REPORTE WHERE _TOP_LEVEL_AURI =
uuid:9e03e883-84bd-4457-a9cd-ae54746e1139 AND _PARENT_AURI =
uuid:9e03e883-84bd-4457-a9cd-ae54746e1139 multiple-choice valueSet is
missing an entry OR has an extra copy of one

and then, on the datastore tab,

entering in the query box:

SELECT * FROM opendatakit.MAPFRE_JUNIO2016_G3_REPORTE WHERE
_TOP_LEVEL_AURI = "uuid:9e03e883-84bd-4457-a9cd-ae54746e1139" AND
_PARENT_AURI = "uuid:9e03e883-84bd-4457-a9cd-ae54746e1139"

you'll find 2 entries with matching _ORDINAL_NUMBER values (1).

Per the instructions (step 6 & 7), delete the older one of these.

On Mon, Jun 6, 2016 at 9:39 AM, Pablo Rodríguez < nuncaestardes...@gmail.com> wrote:

Well Now in another project I´m having the very same issue. wort
timing ever. In the middle of a big running project :S

There are not incomplete submissions according to ODK.

I was publishing a google spreadsheet and a fusion table (Running in
google app engine with billing enabled)

I´m looking at the log and I´m having an average of 4 to 7 error per
minute, basically like this:

18:14:55.688GET500956 B550 msAppEngine-Google; (+
http://code.google.com/appengine)/gae/uploadSubmissionsTask?fscUri=uuid%3A9672732c-1bc6-4dde-a463-d066b5f3edf3

0.1.0.2 - - [06/Jun/2016:18:14:55 +0200] "GET
/gae/uploadSubmissionsTask?fscUri=uuid%3A9672732c-1bc6-4dde-a463-d066b5f3edf3
HTTP/1.1" 500 956
http://olmedo-servicios-integrales.appspot.com/gae/watchdog
"AppEngine-Google; (+http://code.google.com/appengine)" "
olmedo-servicios-integrales.appspot.com" ms=550 cpu_ms=231
cpm_usd=2.5548e-7 loading_request=0
instance=00c61b117c327d751511c82e8ba427e6081789412cae8a6b2a33f690566e8b31
app_engine_release=1.9.38 trace_id=-

{

metadata: {…}
protoPayload: {…}
insertId: "2016-06-06|09:15:00.884913-07|10.106.162.86|1409825575"
log: "appengine.googleapis.com/request_log"
httpRequest: {…}
operation: {…} }

Seaching for OutOfMemory in the log and I don´t find anything.

Right now I just pulled all the information with Briefcase, and As
far as I can see I have more than what I had in my published tables.

My biggest concern is that I could loose submissions :S.

Any help is very appreciated.

Regards,

On Fri, May 6, 2016 at 6:22 PM, Mitch Sundt mitche...@gmail.com wrote:

This would indicate a data corruption issue.

See these instructions:
https://github.com/opendatakit/opendatakit/wiki/Aggregate-AppEngine-Troubleshooting#reparing-a-filled-in-form-submission

w.r.t. ODK Briefcase starting to pull data from the beginning every
time, if, after you pull data, you issue a push back up to the same server,
this will set tracking flags such that those records will be ignored and
skipped over the next time you pull.

ODK Briefcase stops at the first corrupted record because it is
fetching data in ascending marked-as-complete timestamp order. Until the
corrupted record is repaired or removed, it will not proceed to other
records.

Data corruption is more likely to occur if your server memory is
becoming exhausted or if you are submitting data over very-low-bandwidth
communications channels (e.g., satellite).

On Fri, May 6, 2016 at 5:52 AM, nuncaestardes...@gmail.com wrote:

El viernes, 6 de mayo de 2016, 12:30:39 (UTC+2), nuncaestardes...@gmail.com escribió:

El martes, 3 de mayo de 2016, 14:10:28 (UTC+2), nuncaestardes...@gmail.com escribió:

Hi guys,

I´m kind of new here, but I already have this problem:

On ODK Aggregate, when I want to export my data to .csv file,
It´s showing me an error (Failure - will retry later). it was working fine
but I guess the DDBB reach a certain volume that made it fail.

I have read that it is related to global max_allowed_packet
which has to be set to a higher value, with the following expression

set global max_allowed_packet = 1073741824;

I have my ODK aggregate deployed in a google app engine, so i
tied to type that in the cloud shell but nothing happened, I still have the
same problem, and as you can imaging it´s the worst moment to happend...

I have a excel macro that reads the csv file and download the
media files, renames and save everything in a file, so then i can open each
picture from a link in an excel spreadsheet.

Now I can only download the the CSV using ODK briefcase by:

1- Pulling all the data.

2- Exporting the CSV (I can only do it if I firstly pull the
data)

The problem is that by doing this I have to download the media
files twice (in briefcase and in my macro), and the CSV file exported by
briefcase contains the local link in my computer to the media files, but
not the cloud link to the media files, which in my case it is super
convenient. I figure out that the link to the media is a concatenation of
the ID number and some other things, so I manage to bypass this problem,
but I had to change my whole procedure in the middle of the work, making it
much more manual and not allowing me to use the tools I developped to
automate all this work.

Anyone knows how can I in ODK aggregate deployed in google app
engine execute this procedure?

set global max_allowed_packet = 1073741824;

IF that really is the problem, which i hope it is.

Thank you very much in advance.

Regards,

Pablo

One more thing, everytime I pull information with briefcase it
downloads the whole project instead of just downloading the new
entries,that could be something easy to improve in future versions, my
excel macro that downloads everything from the csv does that it is muc more
convenient this way.

Regards,

Pablo

I am trying to pull all the information with briefcase, and looking
at the log I have this:

...

fetching instance 3048 ...

fetching instance 3049 ...

fetching instance 3050 ...

fetching instance 3051 ...

fetching instance 3052 ...

fetching instance 3053 ...

retrieving next chunk of instances from server...

FAILED.

I don´t know if it failed because there is nothing more to pull or
because actually there is something elses happening... right now I don´t
know how many entries I have so 3053? I don´t know... the filter in
aggregate does not work so i don´t know how could I do this...

--

--

Post: opend...@googlegroups.com

Unsubscribe: opendatakit...@googlegroups.com

Options: http://groups.google.com/group/opendatakit?hl=en

You received this message because you are subscribed to the Google
Groups "ODK Community" group.

To unsubscribe from this group and stop receiving emails from it,
send an email to opendatakit...@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--

Mitch Sundt
Software Engineer
University of Washington
mitche...@gmail.com

--

--

Post: opend...@googlegroups.com

Unsubscribe: opendatakit...@googlegroups.com

Options: http://groups.google.com/group/opendatakit?hl=en

You received this message because you are subscribed to a topic in
the Google Groups "ODK Community" group.

To unsubscribe from this topic, visit
https://groups.google.com/d/topic/opendatakit/hjMKcHkk1tc/unsubscribe.

To unsubscribe from this group and all its topics, send an email to
opendatakit...@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--

--

Post: opend...@googlegroups.com

Unsubscribe: opendatakit...@googlegroups.com

Options: http://groups.google.com/group/opendatakit?hl=en

You received this message because you are subscribed to the Google
Groups "ODK Community" group.

To unsubscribe from this group and stop receiving emails from it,
send an email to opendatakit...@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--

Mitch Sundt
Software Engineer
University of Washington
mitche...@gmail.com

--

Post: opendatakit@googlegroups.com
Unsubscribe: opendatakit+unsubscribe@googlegroups.com
Options: http://groups.google.com/group/opendatakit?hl=en

You received this message because you are subscribed to a topic in the
Google Groups "ODK Community" group.
To unsubscribe from this topic, visit
https://groups.google.com/d/topic/opendatakit/hjMKcHkk1tc/unsubscribe.
To unsubscribe from this group and all its topics, send an email to
opendatakit+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--

Post: opendatakit@googlegroups.com
Unsubscribe: opendatakit+unsubscribe@googlegroups.com
Options: http://groups.google.com/group/opendatakit?hl=en

You received this message because you are subscribed to the Google Groups
"ODK Community" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to opendatakit+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--
Mitch Sundt
Software Engineer
University of Washington
mitchellsundt@gmail.com

Pablo_Rodriguez1 · June 8, 2016, 7:30am

IT WORKED!!!!

thank you very much!!!!!

the publishing started automatically, I guess that the 5-7 errors I was having per second was due to trying to publish.

I am extremely happy, not only for resolving this issue but for understanding it. Now I think I could do it again

Mitch you are the best!

···

El martes, 7 de junio de 2016, 23:26:31 (UTC+2), Mitch Sundt escribió: > You are very close. Note that in step (5), the instructions say: > > > > 5) Now, for each data table name and XXXXX value in step (2), execute a query of the form:SELECT * FROM `opendatakit.YOUR_TABLE_NAME` WHERE _PARENT_AURI = "XXXXXX" > > Take special note of the backquote around the opendatakit.-qualified table name, and the double-quotes around the XXXXXX value. If the original query specified _DOM_AURI instead of_PARENT_AURI, use that. If this query does not return a value, you may also need to add the AND _TOP_LEVEL_AURI = "YYYYY" qualifier (but that should not be necessary), if it was present in the original error message. > > > In the queries you tried, you missed this part of the instructions (replace MAPFRE_JUNIO2016_G2_FOTO1_BN in your select statements with `opendatakit.MAPFRE_JUNIO2016_G2_FOTO1_BN` ) > > > These instructions are saying to select and show to you the entries in the data tables that correspond to a specific submission (the one with instanceID uuid:9e03e883-84bd-4457-a9cd-ae54746e1139 ). > > > ======================== > If the publisher is paused, I believe the "PAUSED" is actually a button, and you can click on that to get it to resume. > > > That, of course, will only be successful if you fix the data corruption. > > > > > On Tue, Jun 7, 2016 at 10:05 AM, Pablo Rodríguez wrote: > > One last thing and hoping I can fix the main problem, since this is secondary: I tried to publish another spreadsheet (didnt´t work) and another fusion table (just existing submissions, 1420, when I had already published 1476 in the original) again, but at the end I decided to delete them. The two original spreadsheet and fusion table are both "BOTH Upload Existing & Stream New Submission Data" but now the status shows "PAUSED". How can I make them active again? > > > > > On Tue, Jun 7, 2016 at 5:32 PM, Pablo Rodríguez wrote: > > Ok, I think I´m seeing some progress in the sense of figuring out, what´s happening. I see where did you took the query you gave me, from Logs, App Engine, Background, All versions: > > > > If I enlarge the first error I can see what was happening: > > > > > And I followed your instructions. But there is a second error, and it is different: > > > where it says: > > > org.opendatakit.aggregate.task.CsvWorkerImpl failureRecovery: Exception caught: org.opendatakit.common.datamodel.ODKEnumeratedElementException: Attachment errors:SELECT * FROM MAPFRE_JUNIO2016_G2_FOTO1_BN WHERE _TOP_LEVEL_AURI = uuid:9e03e883-84bd-4457-a9cd-ae54746e1139 AND _PARENT_AURI = uuid:9e03e883-84bd-4457-a9cd-ae54746e1139 is missing an attachment instance OR has extra copies. for MapfreJunio2016 > (BTW, in MAPFRE_JUNIO2016_G2_FOTO1_BN G2 stands for group1 since in this form there are several fields displayed in the same screen, and FOTO1 means "picture1" which is a required field... I don´t know if that is relevant) > > > Then I decided to query: > > > SELECT * FROM MAPFRE_JUNIO2016_G2_FOTO1_BN WHERE _TOP_LEVEL_AURI = "uuid:9e03e883-84bd-4457-a9cd-ae54746e1139" AND _PARENT_AURI = "uuid:9e03e883-84bd-4457-a9cd-ae54746e1139" > > > > But in this case instead of finding 2 entities there is no match. Also I can see that by searching this folder in the instances folder created by ODK Briefcase, instead of having an XML file and a picture/s it is empty. > > > So I guess now I should find a way of deleting uuid:9e03e883-84bd-4457-a9cd-ae54746e1139 in the database since it is a void submission, but I don´t know how to do this. ¿how can I do this? > > > What about all the rest of errors that I am finding in APP Engine,Default Service that I posted before? should I worry about them? (Because I do) > > > Why is this happening? How can I prevent this to happen again? could large pictures be a problem? (right now the pulled process made by briefcase is 4,24GB which includes XML files and pictures. This is 65% of what I am expecting) > > > Thank you very much in advance. > > > > > > > > > > > On Tue, Jun 7, 2016 at 10:09 AM, Pablo Rodríguez wrote: > > Ok So I see there are two errors constantly happening: > > > > > one of them: > > > > > > > 09:55:58.212GET500960 B5.9 sAppEngine-Google; (+http://code.google.com/appengine)/gae/uploadSubmissionsTask?fscUri=uuid%3A9672732c-1bc6-4dde-a463-d066b5f3edf3 > > > 0.1.0.2 - - [07/Jun/2016:09:55:58 +0200] "GET /gae/uploadSubmissionsTask?fscUri=uuid%3A9672732c-1bc6-4dde-a463-d066b5f3edf3 HTTP/1.1" 500 960 http://olmedo-servicios-integrales.appspot.com/gae/watchdog "AppEngine-Google; (+http://code.google.com/appengine)" "olmedo-servicios-integrales.appspot.com" ms=5901 cpu_ms=460 cpm_usd=3.1940599999999997e-7 loading_request=0 instance=00c61b117c0e9e3d18748ca60cc00676c90a86e4344dfcefb7184ce68bcd8de904 app_engine_release=1.9.38 trace_id=- > > > { > > metadata: {…} > protoPayload: {…} > insertId: "2016-06-07|00:56:08.408633-07|10.94.34.134|-1592936713" > log: "appengine.googleapis.com/request_log" > httpRequest: {…} > operation: {…} } > > 09:56:01.221org.opendatakit.aggregate.task.gae.servlet.UploadSubmissionsTaskServlet doGet: Beginning servlet processing > > 09:56:01.227org.opendatakit.aggregate.task.gae.servlet.UploadSubmissionsTaskServlet doGet: Request is running on frontend > > 09:56:01.228org.opendatakit.aggregate.task.UploadSubmissionsWorkerImpl uploadAllSubmissions: Beginning UPLOAD service: uuid:8979e040-b496-49c1-b8de-f355d69d4c46 form MapfreJunio2016 > > 09:56:01.228org.opendatakit.aggregate.form.FormFactory internalGetForms: FormCache: using cached list of Forms > > 09:56:02.449org.opendatakit.aggregate.form.FormFactory internalGetForms: FormCache: using cached list of Forms > > 09:56:02.478[s~olmedo-servicios-integrales/1.392851787586356529].: Time left on lock: -73233127 > > > 09:56:02.478[s~olmedo-servicios-integrales/1.392851787586356529].: Time left on lock: -13406599 > > > 09:56:02.478[s~olmedo-servicios-integrales/1.392851787586356529].: Time left on lock: -52076838 > > > 09:56:02.478[s~olmedo-servicios-integrales/1.392851787586356529].: Time left on lock: -57872843 > > > 09:56:02.478[s~olmedo-servicios-integrales/1.392851787586356529].: Time left on lock: -78707824 > > > And the other one > > > > > > > 09:56:03.646GET2020 B1.4 sAppEngine-Google; (+http://code.google.com/appengine)/gae/uploadSubmissionsTask?fscUri=uuid%3A7ad28143-2a7a-4624-8807-215f05fd0102 > > > 0.1.0.2 - - [07/Jun/2016:09:56:03 +0200] "GET /gae/uploadSubmissionsTask?fscUri=uuid%3A7ad28143-2a7a-4624-8807-215f05fd0102 HTTP/1.1" 202 - http://olmedo-servicios-integrales.appspot.com/gae/watchdog "AppEngine-Google; (+http://code.google.com/appengine)" "olmedo-servicios-integrales.appspot.com" ms=1400 cpu_ms=120 cpm_usd=1.39586e-7 loading_request=0 instance=00c61b117c0e9e3d18748ca60cc00676c90a86e4344dfcefb7184ce68bcd8de904 app_engine_release=1.9.38 trace_id=- > > > { > > metadata: {…} > protoPayload: {…} > insertId: "2016-06-07|00:56:08.408751-07|10.94.34.134|-1592936713" > log: "appengine.googleapis.com/request_log" > httpRequest: {…} > operation: {…} } > > 09:56:03.652org.opendatakit.aggregate.task.gae.servlet.UploadSubmissionsTaskServlet doGet: Beginning servlet processing > > 09:56:03.658org.opendatakit.aggregate.task.gae.servlet.UploadSubmissionsTaskServlet doGet: Request is running on frontend > > 09:56:03.658org.opendatakit.aggregate.task.UploadSubmissionsWorkerImpl uploadAllSubmissions: Beginning UPLOAD service: uuid:28dbf50b-2fcf-4424-800f-9c90eb823a09 form MapfreJunio2016 > > 09:56:03.658org.opendatakit.aggregate.form.FormFactory internalGetForms: FormCache: using cached list of Forms > > 09:56:03.856org.opendatakit.aggregate.form.FormFactory internalGetForms: FormCache: using cached list of Forms > > 09:56:03.881[s~olmedo-servicios-integrales/1.392851787586356529].: Time left on lock: -14278749 > > > 09:56:03.881[s~olmedo-servicios-integrales/1.392851787586356529].: Time left on lock: -17503047 > > > 09:56:03.881[s~olmedo-servicios-integrales/1.392851787586356529].: Time left on lock: -34874322 > > > 09:56:03.881[s~olmedo-servicios-integrales/1.392851787586356529].: Time left on lock: -50316469 > > > 09:56:03.881[s~olmedo-servicios-integrales/1.392851787586356529].: Time left on lock: -18963331 > > > > > > So I try to run your query but changing the uuid to see if i can see two entries like the first thing I did but it says no entities matched. I checked and the publishing is still not working, but when I pull with Briefcase I can see it is downloading more instances, so I guess the system keeps receiving forms... > > > > > > > On Tue, Jun 7, 2016 at 9:51 AM, Pablo Rodríguez wrote: > > > I´ve done exactly what do told me, now I´m pulling with briefcase to see if it works, but ends with FAILED > > > > > The problem is that I´m struggling to understand what I am doing, so the steps for Repairing a Filled-in Form Submission are not straight forward for me. I´ll keep trying... > > > > > > > > > On Tue, Jun 7, 2016 at 9:22 AM, wrote: > I´m going to try Mitch, for me this is dancing on thin ice... let´s see...! > > > > Thanks! > > > > > > El martes, 7 de junio de 2016, 0:55:47 (UTC+2), Mitch Sundt escribió: > > > Have you worked through the database repair steps described here: > > > > > > > > > https://github.com/opendatakit/opendatakit/wiki/Aggregate-AppEngine-Troubleshooting#repairing-a-filled-in-form-submission > > > > > > > > > I.e., > > > > > > > > > org.opendatakit.aggregate.task.gae.servlet.UploadSubmissionsTaskServlet doGet: org.opendatakit.aggregate.exception.ODKExternalServiceException: org.opendatakit.common.datamodel.ODKEnumeratedElementException: SELECT * FROM MAPFRE_JUNIO2016_G3_REPORTE WHERE _TOP_LEVEL_AURI = uuid:9e03e883-84bd-4457-a9cd-ae54746e1139 AND _PARENT_AURI = uuid:9e03e883-84bd-4457-a9cd-ae54746e1139 multiple-choice valueSet is missing an entry OR has an extra copy of one > > > > > > > > > > > > > > > and then, on the datastore tab, > > > > > > > > > entering in the query box: > > > > > > > > > > > > SELECT * FROM `opendatakit.MAPFRE_JUNIO2016_G3_REPORTE` WHERE _TOP_LEVEL_AURI = "uuid:9e03e883-84bd-4457-a9cd-ae54746e1139" AND _PARENT_AURI = "uuid:9e03e883-84bd-4457-a9cd-ae54746e1139" > > > > > > > > > you'll find 2 entries with matching _ORDINAL_NUMBER values (1). > > > > > > > > > Per the instructions (step 6 & 7), delete the older one of these. > > > > > > > > > > > > > > > On Mon, Jun 6, 2016 at 9:39 AM, Pablo Rodríguez wrote: > > > > > > Well Now in another project I´m having the very same issue. wort timing ever. In the middle of a big running project :S > > > > > > > > > There are not incomplete submissions according to ODK. > > > > > > > > > > > > > > > > > > > > > I was publishing a google spreadsheet and a fusion table (Running in google app engine with billing enabled) > > > > > > > > > > > > > > > I´m looking at the log and I´m having an average of 4 to 7 error per minute, basically like this: > > > > > > > > > > > > > > > > > > 18:14:55.688GET500956 B550 msAppEngine-Google; (+http://code.google.com/appengine)/gae/uploadSubmissionsTask?fscUri=uuid%3A9672732c-1bc6-4dde-a463-d066b5f3edf3 > > > > > > > > > 0.1.0.2 - - [06/Jun/2016:18:14:55 +0200] "GET /gae/uploadSubmissionsTask?fscUri=uuid%3A9672732c-1bc6-4dde-a463-d066b5f3edf3 HTTP/1.1" 500 956 http://olmedo-servicios-integrales.appspot.com/gae/watchdog "AppEngine-Google; (+http://code.google.com/appengine)" "olmedo-servicios-integrales.appspot.com" ms=550 cpu_ms=231 cpm_usd=2.5548e-7 loading_request=0 instance=00c61b117c327d751511c82e8ba427e6081789412cae8a6b2a33f690566e8b31 app_engine_release=1.9.38 trace_id=- > > > > > > > > > { > > > > > > metadata: {…} > > > protoPayload: {…} > > > insertId: "2016-06-06|09:15:00.884913-07|10.106.162.86|1409825575" > > > log: "appengine.googleapis.com/request_log" > > > httpRequest: {…} > > > operation: {…} } > > > > > > > > > Seaching for OutOfMemory in the log and I don´t find anything. > > > > > > > > > Right now I just pulled all the information with Briefcase, and As far as I can see I have more than what I had in my published tables. > > > > > > > > > My biggest concern is that I could loose submissions :S. > > > > > > > > > Any help is very appreciated. > > > > > > > > > Regards, > > > > > > > > > > > > > > > On Fri, May 6, 2016 at 6:22 PM, Mitch Sundt wrote: > > > > > > > > > > > > This would indicate a data corruption issue. > > > > > > > > > See these instructions: https://github.com/opendatakit/opendatakit/wiki/Aggregate-AppEngine-Troubleshooting#reparing-a-filled-in-form-submission > > > > > > > > > w.r.t. ODK Briefcase starting to pull data from the beginning every time, if, after you pull data, you issue a push back up to the same server, this will set tracking flags such that those records will be ignored and skipped over the next time you pull. > > > > > > > > > ODK Briefcase stops at the first corrupted record because it is fetching data in ascending marked-as-complete timestamp order. Until the corrupted record is repaired or removed, it will not proceed to other records. > > > > > > > > > Data corruption is more likely to occur if your server memory is becoming exhausted or if you are submitting data over very-low-bandwidth communications channels (e.g., satellite). > > > > > > > > > > > > > > > > > > > > > > > On Fri, May 6, 2016 at 5:52 AM, wrote: > > > > > > > > > El viernes, 6 de mayo de 2016, 12:30:39 (UTC+2), nuncaestardes...@gmail.com escribió: > > > > > > > El martes, 3 de mayo de 2016, 14:10:28 (UTC+2), nuncaestardes...@gmail.com escribió: > > > > > > > > Hi guys, > > > > > > > > > > > > > > > > I´m kind of new here, but I already have this problem: > > > > > > > > > > > > > > > > On ODK Aggregate, when I want to export my data to .csv file, It´s showing me an error (Failure - will retry later). it was working fine but I guess the DDBB reach a certain volume that made it fail. > > > > > > > > > > > > > > > > I have read that it is related to global max_allowed_packet which has to be set to a higher value, with the following expression > > > > > > > > > > > > > > > > set global max_allowed_packet = 1073741824; > > > > > > > > > > > > > > > > I have my ODK aggregate deployed in a google app engine, so i tied to type that in the cloud shell but nothing happened, I still have the same problem, and as you can imaging it´s the worst moment to happend... > > > > > > > > > > > > > > > > I have a excel macro that reads the csv file and download the media files, renames and save everything in a file, so then i can open each picture from a link in an excel spreadsheet. > > > > > > > > > > > > > > > > Now I can only download the the CSV using ODK briefcase by: > > > > > > > > 1- Pulling all the data. > > > > > > > > 2- Exporting the CSV (I can only do it if I firstly pull the data) > > > > > > > > > > > > > > > > The problem is that by doing this I have to download the media files twice (in briefcase and in my macro), and the CSV file exported by briefcase contains the local link in my computer to the media files, but not the cloud link to the media files, which in my case it is super convenient. I figure out that the link to the media is a concatenation of the ID number and some other things, so I manage to bypass this problem, but I had to change my whole procedure in the middle of the work, making it much more manual and not allowing me to use the tools I developped to automate all this work. > > > > > > > > > > > > > > > > Anyone knows how can I in ODK aggregate deployed in google app engine execute this procedure? > > > > > > > > set global max_allowed_packet = 1073741824; > > > > > > > > > > > > > > > > IF that really is the problem, which i hope it is. > > > > > > > > > > > > > > > > Thank you very much in advance. > > > > > > > > > > > > > > > > Regards, > > > > > > > > > > > > > > > > Pablo > > > > > > > > > > > > > > One more thing, everytime I pull information with briefcase it downloads the whole project instead of just downloading the new entries,that could be something easy to improve in future versions, my excel macro that downloads everything from the csv does that it is muc more convenient this way. > > > > > > > > > > > > > > Regards, > > > > > > > > > > > > > > Pablo > > > > > > > > > > > > I am trying to pull all the information with briefcase, and looking at the log I have this: > > > > > > > > > > > > ... > > > > > > fetching instance 3048 ... > > > > > > fetching instance 3049 ... > > > > > > fetching instance 3050 ... > > > > > > fetching instance 3051 ... > > > > > > fetching instance 3052 ... > > > > > > fetching instance 3053 ... > > > > > > retrieving next chunk of instances from server... > > > > > > FAILED. > > > > > > > > > > > > I don´t know if it failed because there is nothing more to pull or because actually there is something elses happening... right now I don´t know how many entries I have so 3053? I don´t know... the filter in aggregate does not work so i don´t know how could I do this... > > > > > > > > > > > > > > > > > > -- > > > > > > -- > > > > > > Post: opend...@googlegroups.com > > > > > > Unsubscribe: opendatakit...@googlegroups.com > > > > > > Options: http://groups.google.com/group/opendatakit?hl=en > > > > > > > > > > > > --- > > > > > > You received this message because you are subscribed to the Google Groups "ODK Community" group. > > > > > > To unsubscribe from this group and stop receiving emails from it, send an email to opendatakit...@googlegroups.com. > > > > > > For more options, visit https://groups.google.com/d/optout. > > > > > > > > > > > > > > > > > > -- > > > > > > Mitch Sundt > > > Software Engineer > > > University of Washington > > > mitche...@gmail.com > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > -- > > > > > > -- > > > > > > Post: opend...@googlegroups.com > > > > > > Unsubscribe: opendatakit...@googlegroups.com > > > > > > Options: http://groups.google.com/group/opendatakit?hl=en > > > > > > > > > > > > --- > > > > > > You received this message because you are subscribed to a topic in the Google Groups "ODK Community" group. > > > > > > To unsubscribe from this topic, visit https://groups.google.com/d/topic/opendatakit/hjMKcHkk1tc/unsubscribe. > > > > > > To unsubscribe from this group and all its topics, send an email to opendatakit...@googlegroups.com. > > > > > > For more options, visit https://groups.google.com/d/optout. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > -- > > > > > > -- > > > > > > Post: opend...@googlegroups.com > > > > > > Unsubscribe: opendatakit...@googlegroups.com > > > > > > Options: http://groups.google.com/group/opendatakit?hl=en > > > > > > > > > > > > --- > > > > > > You received this message because you are subscribed to the Google Groups "ODK Community" group. > > > > > > To unsubscribe from this group and stop receiving emails from it, send an email to opendatakit...@googlegroups.com. > > > > > > For more options, visit https://groups.google.com/d/optout. > > > > > > > > > > > > > > > > > > -- > > > > > > Mitch Sundt > > > Software Engineer > > > University of Washington > > > mitche...@gmail.com > > > > > > -- > > -- > > Post: opend...@googlegroups.com > > Unsubscribe: opendatakit...@googlegroups.com > > Options: http://groups.google.com/group/opendatakit?hl=en > > > > --- > > You received this message because you are subscribed to a topic in the Google Groups "ODK Community" group. > > To unsubscribe from this topic, visit https://groups.google.com/d/topic/opendatakit/hjMKcHkk1tc/unsubscribe. > > To unsubscribe from this group and all its topics, send an email to opendatakit...@googlegroups.com. > > For more options, visit https://groups.google.com/d/optout. > > > > > > > > > > > > > > -- > > -- > > Post: opend...@googlegroups.com > > Unsubscribe: opendatakit...@googlegroups.com > > Options: http://groups.google.com/group/opendatakit?hl=en > > > > --- > > You received this message because you are subscribed to the Google Groups "ODK Community" group. > > To unsubscribe from this group and stop receiving emails from it, send an email to opendatakit...@googlegroups.com. > > For more options, visit https://groups.google.com/d/optout. > > > > > > -- > > Mitch Sundt > Software Engineer > University of Washington > mitche...@gmail.com

Pablo_Rodriguez1 · June 8, 2016, 8:23am

I´ll complete my answer for the future:

Publishing is restored in both fusion tables and google spreadsheet (significantly slower in the spreadsheet), and now both status is "ACTIVE" (there is no button Mitch)
Briefcase gives me a "SUCCESS" message after pulling the whole thing.
CSV generation works without any problem.

Really this is the best!

···

El miércoles, 8 de junio de 2016, 9:30:55 (UTC+2), tomadedat...@gmail.com escribió: > IT WORKED!!!! > > thank you very much!!!!! > > the publishing started automatically, I guess that the 5-7 errors I was having per second was due to trying to publish. > > I am extremely happy, not only for resolving this issue but for understanding it. Now I think I could do it again :) > > Mitch you are the best! > > El martes, 7 de junio de 2016, 23:26:31 (UTC+2), Mitch Sundt escribió: > > You are very close. Note that in step (5), the instructions say: > > > > > > > > 5) Now, for each data table name and XXXXX value in step (2), execute a query of the form:SELECT * FROM `opendatakit.YOUR_TABLE_NAME` WHERE _PARENT_AURI = "XXXXXX" > > > > Take special note of the backquote around the opendatakit.-qualified table name, and the double-quotes around the XXXXXX value. If the original query specified _DOM_AURI instead of_PARENT_AURI, use that. If this query does not return a value, you may also need to add the AND _TOP_LEVEL_AURI = "YYYYY" qualifier (but that should not be necessary), if it was present in the original error message. > > > > > > In the queries you tried, you missed this part of the instructions (replace MAPFRE_JUNIO2016_G2_FOTO1_BN in your select statements with `opendatakit.MAPFRE_JUNIO2016_G2_FOTO1_BN` ) > > > > > > These instructions are saying to select and show to you the entries in the data tables that correspond to a specific submission (the one with instanceID uuid:9e03e883-84bd-4457-a9cd-ae54746e1139 ). > > > > > > ======================== > > If the publisher is paused, I believe the "PAUSED" is actually a button, and you can click on that to get it to resume. > > > > > > That, of course, will only be successful if you fix the data corruption. > > > > > > > > > > On Tue, Jun 7, 2016 at 10:05 AM, Pablo Rodríguez wrote: > > > > One last thing and hoping I can fix the main problem, since this is secondary: I tried to publish another spreadsheet (didnt´t work) and another fusion table (just existing submissions, 1420, when I had already published 1476 in the original) again, but at the end I decided to delete them. The two original spreadsheet and fusion table are both "BOTH Upload Existing & Stream New Submission Data" but now the status shows "PAUSED". How can I make them active again? > > > > > > > > > > On Tue, Jun 7, 2016 at 5:32 PM, Pablo Rodríguez wrote: > > > > Ok, I think I´m seeing some progress in the sense of figuring out, what´s happening. I see where did you took the query you gave me, from Logs, App Engine, Background, All versions: > > > > > > > > If I enlarge the first error I can see what was happening: > > > > > > > > > > And I followed your instructions. But there is a second error, and it is different: > > > > > > where it says: > > > > > > org.opendatakit.aggregate.task.CsvWorkerImpl failureRecovery: Exception caught: org.opendatakit.common.datamodel.ODKEnumeratedElementException: Attachment errors:SELECT * FROM MAPFRE_JUNIO2016_G2_FOTO1_BN WHERE _TOP_LEVEL_AURI = uuid:9e03e883-84bd-4457-a9cd-ae54746e1139 AND _PARENT_AURI = uuid:9e03e883-84bd-4457-a9cd-ae54746e1139 is missing an attachment instance OR has extra copies. for MapfreJunio2016 > > (BTW, in MAPFRE_JUNIO2016_G2_FOTO1_BN G2 stands for group1 since in this form there are several fields displayed in the same screen, and FOTO1 means "picture1" which is a required field... I don´t know if that is relevant) > > > > > > Then I decided to query: > > > > > > SELECT * FROM MAPFRE_JUNIO2016_G2_FOTO1_BN WHERE _TOP_LEVEL_AURI = "uuid:9e03e883-84bd-4457-a9cd-ae54746e1139" AND _PARENT_AURI = "uuid:9e03e883-84bd-4457-a9cd-ae54746e1139" > > > > > > > > But in this case instead of finding 2 entities there is no match. Also I can see that by searching this folder in the instances folder created by ODK Briefcase, instead of having an XML file and a picture/s it is empty. > > > > > > So I guess now I should find a way of deleting uuid:9e03e883-84bd-4457-a9cd-ae54746e1139 in the database since it is a void submission, but I don´t know how to do this. ¿how can I do this? > > > > > > What about all the rest of errors that I am finding in APP Engine,Default Service that I posted before? should I worry about them? (Because I do) > > > > > > Why is this happening? How can I prevent this to happen again? could large pictures be a problem? (right now the pulled process made by briefcase is 4,24GB which includes XML files and pictures. This is 65% of what I am expecting) > > > > > > Thank you very much in advance. > > > > > > > > > > > > > > > > > > > > > > On Tue, Jun 7, 2016 at 10:09 AM, Pablo Rodríguez wrote: > > > > Ok So I see there are two errors constantly happening: > > > > > > > > > > one of them: > > > > > > > > > > > > > > 09:55:58.212GET500960 B5.9 sAppEngine-Google; (+http://code.google.com/appengine)/gae/uploadSubmissionsTask?fscUri=uuid%3A9672732c-1bc6-4dde-a463-d066b5f3edf3 > > > > > > 0.1.0.2 - - [07/Jun/2016:09:55:58 +0200] "GET /gae/uploadSubmissionsTask?fscUri=uuid%3A9672732c-1bc6-4dde-a463-d066b5f3edf3 HTTP/1.1" 500 960 http://olmedo-servicios-integrales.appspot.com/gae/watchdog "AppEngine-Google; (+http://code.google.com/appengine)" "olmedo-servicios-integrales.appspot.com" ms=5901 cpu_ms=460 cpm_usd=3.1940599999999997e-7 loading_request=0 instance=00c61b117c0e9e3d18748ca60cc00676c90a86e4344dfcefb7184ce68bcd8de904 app_engine_release=1.9.38 trace_id=- > > > > > > { > > > > metadata: {…} > > protoPayload: {…} > > insertId: "2016-06-07|00:56:08.408633-07|10.94.34.134|-1592936713" > > log: "appengine.googleapis.com/request_log" > > httpRequest: {…} > > operation: {…} } > > > > 09:56:01.221org.opendatakit.aggregate.task.gae.servlet.UploadSubmissionsTaskServlet doGet: Beginning servlet processing > > > > 09:56:01.227org.opendatakit.aggregate.task.gae.servlet.UploadSubmissionsTaskServlet doGet: Request is running on frontend > > > > 09:56:01.228org.opendatakit.aggregate.task.UploadSubmissionsWorkerImpl uploadAllSubmissions: Beginning UPLOAD service: uuid:8979e040-b496-49c1-b8de-f355d69d4c46 form MapfreJunio2016 > > > > 09:56:01.228org.opendatakit.aggregate.form.FormFactory internalGetForms: FormCache: using cached list of Forms > > > > 09:56:02.449org.opendatakit.aggregate.form.FormFactory internalGetForms: FormCache: using cached list of Forms > > > > 09:56:02.478[s~olmedo-servicios-integrales/1.392851787586356529].: Time left on lock: -73233127 > > > > > > 09:56:02.478[s~olmedo-servicios-integrales/1.392851787586356529].: Time left on lock: -13406599 > > > > > > 09:56:02.478[s~olmedo-servicios-integrales/1.392851787586356529].: Time left on lock: -52076838 > > > > > > 09:56:02.478[s~olmedo-servicios-integrales/1.392851787586356529].: Time left on lock: -57872843 > > > > > > 09:56:02.478[s~olmedo-servicios-integrales/1.392851787586356529].: Time left on lock: -78707824 > > > > > > And the other one > > > > > > > > > > > > > > 09:56:03.646GET2020 B1.4 sAppEngine-Google; (+http://code.google.com/appengine)/gae/uploadSubmissionsTask?fscUri=uuid%3A7ad28143-2a7a-4624-8807-215f05fd0102 > > > > > > 0.1.0.2 - - [07/Jun/2016:09:56:03 +0200] "GET /gae/uploadSubmissionsTask?fscUri=uuid%3A7ad28143-2a7a-4624-8807-215f05fd0102 HTTP/1.1" 202 - http://olmedo-servicios-integrales.appspot.com/gae/watchdog "AppEngine-Google; (+http://code.google.com/appengine)" "olmedo-servicios-integrales.appspot.com" ms=1400 cpu_ms=120 cpm_usd=1.39586e-7 loading_request=0 instance=00c61b117c0e9e3d18748ca60cc00676c90a86e4344dfcefb7184ce68bcd8de904 app_engine_release=1.9.38 trace_id=- > > > > > > { > > > > metadata: {…} > > protoPayload: {…} > > insertId: "2016-06-07|00:56:08.408751-07|10.94.34.134|-1592936713" > > log: "appengine.googleapis.com/request_log" > > httpRequest: {…} > > operation: {…} } > > > > 09:56:03.652org.opendatakit.aggregate.task.gae.servlet.UploadSubmissionsTaskServlet doGet: Beginning servlet processing > > > > 09:56:03.658org.opendatakit.aggregate.task.gae.servlet.UploadSubmissionsTaskServlet doGet: Request is running on frontend > > > > 09:56:03.658org.opendatakit.aggregate.task.UploadSubmissionsWorkerImpl uploadAllSubmissions: Beginning UPLOAD service: uuid:28dbf50b-2fcf-4424-800f-9c90eb823a09 form MapfreJunio2016 > > > > 09:56:03.658org.opendatakit.aggregate.form.FormFactory internalGetForms: FormCache: using cached list of Forms > > > > 09:56:03.856org.opendatakit.aggregate.form.FormFactory internalGetForms: FormCache: using cached list of Forms > > > > 09:56:03.881[s~olmedo-servicios-integrales/1.392851787586356529].: Time left on lock: -14278749 > > > > > > 09:56:03.881[s~olmedo-servicios-integrales/1.392851787586356529].: Time left on lock: -17503047 > > > > > > 09:56:03.881[s~olmedo-servicios-integrales/1.392851787586356529].: Time left on lock: -34874322 > > > > > > 09:56:03.881[s~olmedo-servicios-integrales/1.392851787586356529].: Time left on lock: -50316469 > > > > > > 09:56:03.881[s~olmedo-servicios-integrales/1.392851787586356529].: Time left on lock: -18963331 > > > > > > > > > > > > So I try to run your query but changing the uuid to see if i can see two entries like the first thing I did but it says no entities matched. I checked and the publishing is still not working, but when I pull with Briefcase I can see it is downloading more instances, so I guess the system keeps receiving forms... > > > > > > > > > > > > > > On Tue, Jun 7, 2016 at 9:51 AM, Pablo Rodríguez wrote: > > > > > > I´ve done exactly what do told me, now I´m pulling with briefcase to see if it works, but ends with FAILED > > > > > > > > > > The problem is that I´m struggling to understand what I am doing, so the steps for Repairing a Filled-in Form Submission are not straight forward for me. I´ll keep trying... > > > > > > > > > > > > > > > > > > On Tue, Jun 7, 2016 at 9:22 AM, wrote: > > I´m going to try Mitch, for me this is dancing on thin ice... let´s see...! > > > > > > > > Thanks! > > > > > > > > > > > > El martes, 7 de junio de 2016, 0:55:47 (UTC+2), Mitch Sundt escribió: > > > > > Have you worked through the database repair steps described here: > > > > > > > > > > > > > > > https://github.com/opendatakit/opendatakit/wiki/Aggregate-AppEngine-Troubleshooting#repairing-a-filled-in-form-submission > > > > > > > > > > > > > > > I.e., > > > > > > > > > > > > > > > org.opendatakit.aggregate.task.gae.servlet.UploadSubmissionsTaskServlet doGet: org.opendatakit.aggregate.exception.ODKExternalServiceException: org.opendatakit.common.datamodel.ODKEnumeratedElementException: SELECT * FROM MAPFRE_JUNIO2016_G3_REPORTE WHERE _TOP_LEVEL_AURI = uuid:9e03e883-84bd-4457-a9cd-ae54746e1139 AND _PARENT_AURI = uuid:9e03e883-84bd-4457-a9cd-ae54746e1139 multiple-choice valueSet is missing an entry OR has an extra copy of one > > > > > > > > > > > > > > > > > > > > > > > > > and then, on the datastore tab, > > > > > > > > > > > > > > > entering in the query box: > > > > > > > > > > > > > > > > > > > > SELECT * FROM `opendatakit.MAPFRE_JUNIO2016_G3_REPORTE` WHERE _TOP_LEVEL_AURI = "uuid:9e03e883-84bd-4457-a9cd-ae54746e1139" AND _PARENT_AURI = "uuid:9e03e883-84bd-4457-a9cd-ae54746e1139" > > > > > > > > > > > > > > > you'll find 2 entries with matching _ORDINAL_NUMBER values (1). > > > > > > > > > > > > > > > Per the instructions (step 6 & 7), delete the older one of these. > > > > > > > > > > > > > > > > > > > > > > > > > On Mon, Jun 6, 2016 at 9:39 AM, Pablo Rodríguez wrote: > > > > > > > > > > Well Now in another project I´m having the very same issue. wort timing ever. In the middle of a big running project :S > > > > > > > > > > > > > > > There are not incomplete submissions according to ODK. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > I was publishing a google spreadsheet and a fusion table (Running in google app engine with billing enabled) > > > > > > > > > > > > > > > > > > > > > > > > > I´m looking at the log and I´m having an average of 4 to 7 error per minute, basically like this: > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > 18:14:55.688GET500956 B550 msAppEngine-Google; (+http://code.google.com/appengine)/gae/uploadSubmissionsTask?fscUri=uuid%3A9672732c-1bc6-4dde-a463-d066b5f3edf3 > > > > > > > > > > > > > > > 0.1.0.2 - - [06/Jun/2016:18:14:55 +0200] "GET /gae/uploadSubmissionsTask?fscUri=uuid%3A9672732c-1bc6-4dde-a463-d066b5f3edf3 HTTP/1.1" 500 956 http://olmedo-servicios-integrales.appspot.com/gae/watchdog "AppEngine-Google; (+http://code.google.com/appengine)" "olmedo-servicios-integrales.appspot.com" ms=550 cpu_ms=231 cpm_usd=2.5548e-7 loading_request=0 instance=00c61b117c327d751511c82e8ba427e6081789412cae8a6b2a33f690566e8b31 app_engine_release=1.9.38 trace_id=- > > > > > > > > > > > > > > > { > > > > > > > > > > metadata: {…} > > > > > protoPayload: {…} > > > > > insertId: "2016-06-06|09:15:00.884913-07|10.106.162.86|1409825575" > > > > > log: "appengine.googleapis.com/request_log" > > > > > httpRequest: {…} > > > > > operation: {…} } > > > > > > > > > > > > > > > Seaching for OutOfMemory in the log and I don´t find anything. > > > > > > > > > > > > > > > Right now I just pulled all the information with Briefcase, and As far as I can see I have more than what I had in my published tables. > > > > > > > > > > > > > > > My biggest concern is that I could loose submissions :S. > > > > > > > > > > > > > > > Any help is very appreciated. > > > > > > > > > > > > > > > Regards, > > > > > > > > > > > > > > > > > > > > > > > > > On Fri, May 6, 2016 at 6:22 PM, Mitch Sundt wrote: > > > > > > > > > > > > > > > > > > > > This would indicate a data corruption issue. > > > > > > > > > > > > > > > See these instructions: https://github.com/opendatakit/opendatakit/wiki/Aggregate-AppEngine-Troubleshooting#reparing-a-filled-in-form-submission > > > > > > > > > > > > > > > w.r.t. ODK Briefcase starting to pull data from the beginning every time, if, after you pull data, you issue a push back up to the same server, this will set tracking flags such that those records will be ignored and skipped over the next time you pull. > > > > > > > > > > > > > > > ODK Briefcase stops at the first corrupted record because it is fetching data in ascending marked-as-complete timestamp order. Until the corrupted record is repaired or removed, it will not proceed to other records. > > > > > > > > > > > > > > > Data corruption is more likely to occur if your server memory is becoming exhausted or if you are submitting data over very-low-bandwidth communications channels (e.g., satellite). > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > On Fri, May 6, 2016 at 5:52 AM, wrote: > > > > > > > > > > > > > > > El viernes, 6 de mayo de 2016, 12:30:39 (UTC+2), nuncaestardes...@gmail.com escribió: > > > > > > > > > > > El martes, 3 de mayo de 2016, 14:10:28 (UTC+2), nuncaestardes...@gmail.com escribió: > > > > > > > > > > > > Hi guys, > > > > > > > > > > > > > > > > > > > > > > > > I´m kind of new here, but I already have this problem: > > > > > > > > > > > > > > > > > > > > > > > > On ODK Aggregate, when I want to export my data to .csv file, It´s showing me an error (Failure - will retry later). it was working fine but I guess the DDBB reach a certain volume that made it fail. > > > > > > > > > > > > > > > > > > > > > > > > I have read that it is related to global max_allowed_packet which has to be set to a higher value, with the following expression > > > > > > > > > > > > > > > > > > > > > > > > set global max_allowed_packet = 1073741824; > > > > > > > > > > > > > > > > > > > > > > > > I have my ODK aggregate deployed in a google app engine, so i tied to type that in the cloud shell but nothing happened, I still have the same problem, and as you can imaging it´s the worst moment to happend... > > > > > > > > > > > > > > > > > > > > > > > > I have a excel macro that reads the csv file and download the media files, renames and save everything in a file, so then i can open each picture from a link in an excel spreadsheet. > > > > > > > > > > > > > > > > > > > > > > > > Now I can only download the the CSV using ODK briefcase by: > > > > > > > > > > > > 1- Pulling all the data. > > > > > > > > > > > > 2- Exporting the CSV (I can only do it if I firstly pull the data) > > > > > > > > > > > > > > > > > > > > > > > > The problem is that by doing this I have to download the media files twice (in briefcase and in my macro), and the CSV file exported by briefcase contains the local link in my computer to the media files, but not the cloud link to the media files, which in my case it is super convenient. I figure out that the link to the media is a concatenation of the ID number and some other things, so I manage to bypass this problem, but I had to change my whole procedure in the middle of the work, making it much more manual and not allowing me to use the tools I developped to automate all this work. > > > > > > > > > > > > > > > > > > > > > > > > Anyone knows how can I in ODK aggregate deployed in google app engine execute this procedure? > > > > > > > > > > > > set global max_allowed_packet = 1073741824; > > > > > > > > > > > > > > > > > > > > > > > > IF that really is the problem, which i hope it is. > > > > > > > > > > > > > > > > > > > > > > > > Thank you very much in advance. > > > > > > > > > > > > > > > > > > > > > > > > Regards, > > > > > > > > > > > > > > > > > > > > > > > > Pablo > > > > > > > > > > > > > > > > > > > > > > One more thing, everytime I pull information with briefcase it downloads the whole project instead of just downloading the new entries,that could be something easy to improve in future versions, my excel macro that downloads everything from the csv does that it is muc more convenient this way. > > > > > > > > > > > > > > > > > > > > > > Regards, > > > > > > > > > > > > > > > > > > > > > > Pablo > > > > > > > > > > > > > > > > > > > > I am trying to pull all the information with briefcase, and looking at the log I have this: > > > > > > > > > > > > > > > > > > > > ... > > > > > > > > > > fetching instance 3048 ... > > > > > > > > > > fetching instance 3049 ... > > > > > > > > > > fetching instance 3050 ... > > > > > > > > > > fetching instance 3051 ... > > > > > > > > > > fetching instance 3052 ... > > > > > > > > > > fetching instance 3053 ... > > > > > > > > > > retrieving next chunk of instances from server... > > > > > > > > > > FAILED. > > > > > > > > > > > > > > > > > > > > I don´t know if it failed because there is nothing more to pull or because actually there is something elses happening... right now I don´t know how many entries I have so 3053? I don´t know... the filter in aggregate does not work so i don´t know how could I do this... > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > -- > > > > > > > > > > -- > > > > > > > > > > Post: opend...@googlegroups.com > > > > > > > > > > Unsubscribe: opendatakit...@googlegroups.com > > > > > > > > > > Options: http://groups.google.com/group/opendatakit?hl=en > > > > > > > > > > > > > > > > > > > > --- > > > > > > > > > > You received this message because you are subscribed to the Google Groups "ODK Community" group. > > > > > > > > > > To unsubscribe from this group and stop receiving emails from it, send an email to opendatakit...@googlegroups.com. > > > > > > > > > > For more options, visit https://groups.google.com/d/optout. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > -- > > > > > > > > > > Mitch Sundt > > > > > Software Engineer > > > > > University of Washington > > > > > mitche...@gmail.com > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > -- > > > > > > > > > > -- > > > > > > > > > > Post: opend...@googlegroups.com > > > > > > > > > > Unsubscribe: opendatakit...@googlegroups.com > > > > > > > > > > Options: http://groups.google.com/group/opendatakit?hl=en > > > > > > > > > > > > > > > > > > > > --- > > > > > > > > > > You received this message because you are subscribed to a topic in the Google Groups "ODK Community" group. > > > > > > > > > > To unsubscribe from this topic, visit https://groups.google.com/d/topic/opendatakit/hjMKcHkk1tc/unsubscribe. > > > > > > > > > > To unsubscribe from this group and all its topics, send an email to opendatakit...@googlegroups.com. > > > > > > > > > > For more options, visit https://groups.google.com/d/optout. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > -- > > > > > > > > > > -- > > > > > > > > > > Post: opend...@googlegroups.com > > > > > > > > > > Unsubscribe: opendatakit...@googlegroups.com > > > > > > > > > > Options: http://groups.google.com/group/opendatakit?hl=en > > > > > > > > > > > > > > > > > > > > --- > > > > > > > > > > You received this message because you are subscribed to the Google Groups "ODK Community" group. > > > > > > > > > > To unsubscribe from this group and stop receiving emails from it, send an email to opendatakit...@googlegroups.com. > > > > > > > > > > For more options, visit https://groups.google.com/d/optout. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > -- > > > > > > > > > > Mitch Sundt > > > > > Software Engineer > > > > > University of Washington > > > > > mitche...@gmail.com > > > > > > > > > > > > -- > > > > -- > > > > Post: opend...@googlegroups.com > > > > Unsubscribe: opendatakit...@googlegroups.com > > > > Options: http://groups.google.com/group/opendatakit?hl=en > > > > > > > > --- > > > > You received this message because you are subscribed to a topic in the Google Groups "ODK Community" group. > > > > To unsubscribe from this topic, visit https://groups.google.com/d/topic/opendatakit/hjMKcHkk1tc/unsubscribe. > > > > To unsubscribe from this group and all its topics, send an email to opendatakit...@googlegroups.com. > > > > For more options, visit https://groups.google.com/d/optout. > > > > > > > > > > > > > > > > > > > > > > > > > > > > -- > > > > -- > > > > Post: opend...@googlegroups.com > > > > Unsubscribe: opendatakit...@googlegroups.com > > > > Options: http://groups.google.com/group/opendatakit?hl=en > > > > > > > > --- > > > > You received this message because you are subscribed to the Google Groups "ODK Community" group. > > > > To unsubscribe from this group and stop receiving emails from it, send an email to opendatakit...@googlegroups.com. > > > > For more options, visit https://groups.google.com/d/optout. > > > > > > > > > > > > -- > > > > Mitch Sundt > > Software Engineer > > University of Washington > > mitche...@gmail.com

Mitch_S · June 8, 2016, 5:27pm

Congrats!

I'm glad you worked through the (very-confusing) sequence of steps and were
able to get your system up and running again.

(It's been years since I worked on the PAUSED mechanism -- I was clearly
mistaken about how it functions. I'm glad that it automatically resumed.)

···

On Wed, Jun 8, 2016 at 1:23 AM, wrote:

I´ll complete my answer for the future:

Publishing is restored in both fusion tables and google spreadsheet
(significantly slower in the spreadsheet), and now both status is "ACTIVE"
(there is no button Mitch)

Briefcase gives me a "SUCCESS" message after pulling the whole thing.

CSV generation works without any problem.

Really this is the best!

El miércoles, 8 de junio de 2016, 9:30:55 (UTC+2), tomadedat...@gmail.com escribió:

IT WORKED!!!!

thank you very much!!!!!

the publishing started automatically, I guess that the 5-7 errors I was
having per second was due to trying to publish.

I am extremely happy, not only for resolving this issue but for
understanding it. Now I think I could do it again

Mitch you are the best!

El martes, 7 de junio de 2016, 23:26:31 (UTC+2), Mitch Sundt escribió:

You are very close. Note that in step (5), the instructions say:

Now, for each data table name and XXXXX value in step (2), execute
a query of the form:SELECT * FROM opendatakit.YOUR_TABLE_NAME WHERE
_PARENT_AURI = "XXXXXX"

Take special note of the backquote around the opendatakit.-qualified
table name, and the double-quotes around the XXXXXX value. If the original
query specified _DOM_AURI instead of_PARENT_AURI, use that. If this query
does not return a value, you may also need to add the AND _TOP_LEVEL_AURI =
"YYYYY" qualifier (but that should not be necessary), if it was present in
the original error message.

In the queries you tried, you missed this part of the instructions
(replace MAPFRE_JUNIO2016_G2_FOTO1_BN in your select statements with
opendatakit.MAPFRE_JUNIO2016_G2_FOTO1_BN )

These instructions are saying to select and show to you the entries in
the data tables that correspond to a specific submission (the one with
instanceID uuid:9e03e883-84bd-4457-a9cd-ae54746e1139 ).

========================
If the publisher is paused, I believe the "PAUSED" is actually a
button, and you can click on that to get it to resume.

That, of course, will only be successful if you fix the data
corruption.

On Tue, Jun 7, 2016 at 10:05 AM, Pablo Rodríguez < nuncaestardes...@gmail.com> wrote:

One last thing and hoping I can fix the main problem, since this is
secondary: I tried to publish another spreadsheet (didnt´t work) and
another fusion table (just existing submissions, 1420, when I had already
published 1476 in the original) again, but at the end I decided to delete
them. The two original spreadsheet and fusion table are both "BOTH Upload
Existing & Stream New Submission Data" but now the status shows "PAUSED".
How can I make them active again?

On Tue, Jun 7, 2016 at 5:32 PM, Pablo Rodríguez < nuncaestardes...@gmail.com> wrote:

Ok, I think I´m seeing some progress in the sense of figuring out,
what´s happening. I see where did you took the query you gave me, from
Logs, App Engine, Background, All versions:

If I enlarge the first error I can see what was happening:

And I followed your instructions. But there is a second error, and it
is different:

where it says:

org.opendatakit.aggregate.task.CsvWorkerImpl failureRecovery:
Exception caught:
org.opendatakit.common.datamodel.ODKEnumeratedElementException: Attachment
errors:SELECT * FROM MAPFRE_JUNIO2016_G2_FOTO1_BN WHERE _TOP_LEVEL_AURI =
uuid:9e03e883-84bd-4457-a9cd-ae54746e1139 AND _PARENT_AURI =
uuid:9e03e883-84bd-4457-a9cd-ae54746e1139 is missing an attachment instance
OR has extra copies. for MapfreJunio2016
(BTW, in MAPFRE_JUNIO2016_G2_FOTO1_BN G2 stands for group1 since in
this form there are several fields displayed in the same screen, and FOTO1
means "picture1" which is a required field... I don´t know if that is
relevant)

Then I decided to query:

SELECT * FROM MAPFRE_JUNIO2016_G2_FOTO1_BN WHERE _TOP_LEVEL_AURI =
"uuid:9e03e883-84bd-4457-a9cd-ae54746e1139" AND _PARENT_AURI =
"uuid:9e03e883-84bd-4457-a9cd-ae54746e1139"

But in this case instead of finding 2 entities there is no match. Also
I can see that by searching this folder in the instances folder created by
ODK Briefcase, instead of having an XML file and a picture/s it is empty.

So I guess now I should find a way of
deleting uuid:9e03e883-84bd-4457-a9cd-ae54746e1139 in the database since it
is a void submission, but I don´t know how to do this. ¿how can I do this?

What about all the rest of errors that I am finding in APP
Engine,Default Service that I posted before? should I worry about them?
(Because I do)

Why is this happening? How can I prevent this to happen again? could
large pictures be a problem? (right now the pulled process made by
briefcase is 4,24GB which includes XML files and pictures. This is 65% of
what I am expecting)

Thank you very much in advance.

On Tue, Jun 7, 2016 at 10:09 AM, Pablo Rodríguez < nuncaestardes...@gmail.com> wrote:

Ok So I see there are two errors constantly happening:

one of them:

09:55:58.212GET500960 B5.9 sAppEngine-Google; (+
http://code.google.com/appengine)/gae/uploadSubmissionsTask?fscUri=uuid%3A9672732c-1bc6-4dde-a463-d066b5f3edf3

0.1.0.2 - - [07/Jun/2016:09:55:58 +0200] "GET
/gae/uploadSubmissionsTask?fscUri=uuid%3A9672732c-1bc6-4dde-a463-d066b5f3edf3
HTTP/1.1" 500 960
http://olmedo-servicios-integrales.appspot.com/gae/watchdog
"AppEngine-Google; (+http://code.google.com/appengine)" "
olmedo-servicios-integrales.appspot.com" ms=5901 cpu_ms=460
cpm_usd=3.1940599999999997e-7 loading_request=0
instance=00c61b117c0e9e3d18748ca60cc00676c90a86e4344dfcefb7184ce68bcd8de904
app_engine_release=1.9.38 trace_id=-

{

metadata: {…}
protoPayload: {…}
insertId: "2016-06-07|00:56:08.408633-07|10.94.34.134|-1592936713"
log: "appengine.googleapis.com/request_log"
httpRequest: {…}
operation: {…} }

09:56:01.221org.opendatakit.aggregate.task.gae.servlet.UploadSubmissionsTaskServlet
doGet: Beginning servlet processing

09:56:01.227org.opendatakit.aggregate.task.gae.servlet.UploadSubmissionsTaskServlet
doGet: Request is running on frontend

09:56:01.228org.opendatakit.aggregate.task.UploadSubmissionsWorkerImpl
uploadAllSubmissions: Beginning UPLOAD service:
uuid:8979e040-b496-49c1-b8de-f355d69d4c46 form MapfreJunio2016

09:56:01.228org.opendatakit.aggregate.form.FormFactory
internalGetForms: FormCache: using cached list of Forms

09:56:02.449org.opendatakit.aggregate.form.FormFactory
internalGetForms: FormCache: using cached list of Forms

09:56:02.478[s~olmedo-servicios-integrales/1.392851787586356529].:
Time left on lock: -73233127

09:56:02.478[s~olmedo-servicios-integrales/1.392851787586356529].:
Time left on lock: -13406599

09:56:02.478[s~olmedo-servicios-integrales/1.392851787586356529].:
Time left on lock: -52076838

09:56:02.478[s~olmedo-servicios-integrales/1.392851787586356529].:
Time left on lock: -57872843

09:56:02.478[s~olmedo-servicios-integrales/1.392851787586356529].:
Time left on lock: -78707824

And the other one

09:56:03.646GET2020 B1.4 sAppEngine-Google; (+
http://code.google.com/appengine)/gae/uploadSubmissionsTask?fscUri=uuid%3A7ad28143-2a7a-4624-8807-215f05fd0102

0.1.0.2 - - [07/Jun/2016:09:56:03 +0200] "GET
/gae/uploadSubmissionsTask?fscUri=uuid%3A7ad28143-2a7a-4624-8807-215f05fd0102
HTTP/1.1" 202 -
http://olmedo-servicios-integrales.appspot.com/gae/watchdog
"AppEngine-Google; (+http://code.google.com/appengine)" "
olmedo-servicios-integrales.appspot.com" ms=1400 cpu_ms=120
cpm_usd=1.39586e-7 loading_request=0
instance=00c61b117c0e9e3d18748ca60cc00676c90a86e4344dfcefb7184ce68bcd8de904
app_engine_release=1.9.38 trace_id=-

{

metadata: {…}
protoPayload: {…}
insertId: "2016-06-07|00:56:08.408751-07|10.94.34.134|-1592936713"
log: "appengine.googleapis.com/request_log"
httpRequest: {…}
operation: {…} }

09:56:03.652org.opendatakit.aggregate.task.gae.servlet.UploadSubmissionsTaskServlet
doGet: Beginning servlet processing

09:56:03.658org.opendatakit.aggregate.task.gae.servlet.UploadSubmissionsTaskServlet
doGet: Request is running on frontend

09:56:03.658org.opendatakit.aggregate.task.UploadSubmissionsWorkerImpl
uploadAllSubmissions: Beginning UPLOAD service:
uuid:28dbf50b-2fcf-4424-800f-9c90eb823a09 form MapfreJunio2016

09:56:03.658org.opendatakit.aggregate.form.FormFactory
internalGetForms: FormCache: using cached list of Forms

09:56:03.856org.opendatakit.aggregate.form.FormFactory
internalGetForms: FormCache: using cached list of Forms

09:56:03.881[s~olmedo-servicios-integrales/1.392851787586356529].:
Time left on lock: -14278749

09:56:03.881[s~olmedo-servicios-integrales/1.392851787586356529].:
Time left on lock: -17503047

09:56:03.881[s~olmedo-servicios-integrales/1.392851787586356529].:
Time left on lock: -34874322

09:56:03.881[s~olmedo-servicios-integrales/1.392851787586356529].:
Time left on lock: -50316469

09:56:03.881[s~olmedo-servicios-integrales/1.392851787586356529].:
Time left on lock: -18963331

So I try to run your query but changing the uuid to see if i can see
two entries like the first thing I did but it says no entities matched. I
checked and the publishing is still not working, but when I pull with
Briefcase I can see it is downloading more instances, so I guess the system
keeps receiving forms...

On Tue, Jun 7, 2016 at 9:51 AM, Pablo Rodríguez < nuncaestardes...@gmail.com> wrote:

I´ve done exactly what do told me, now I´m pulling with briefcase to
see if it works, but ends with FAILED

The problem is that I´m struggling to understand what I am doing, so
the steps for Repairing a Filled-in Form Submission are not straight
forward for me. I´ll keep trying...

On Tue, Jun 7, 2016 at 9:22 AM, nuncaestardes...@gmail.com wrote:
I´m going to try Mitch, for me this is dancing on thin ice... let´s
see...!

Thanks!

El martes, 7 de junio de 2016, 0:55:47 (UTC+2), Mitch Sundt escribió:

Have you worked through the database repair steps described here:

https://github.com/opendatakit/opendatakit/wiki/Aggregate-AppEngine-Troubleshooting#repairing-a-filled-in-form-submission

I.e.,

org.opendatakit.aggregate.task.gae.servlet.UploadSubmissionsTaskServlet
doGet: org.opendatakit.aggregate.exception.ODKExternalServiceException:
org.opendatakit.common.datamodel.ODKEnumeratedElementException: SELECT *
FROM MAPFRE_JUNIO2016_G3_REPORTE WHERE _TOP_LEVEL_AURI =
uuid:9e03e883-84bd-4457-a9cd-ae54746e1139 AND _PARENT_AURI =
uuid:9e03e883-84bd-4457-a9cd-ae54746e1139 multiple-choice valueSet is
missing an entry OR has an extra copy of one

and then, on the datastore tab,

entering in the query box:

SELECT * FROM opendatakit.MAPFRE_JUNIO2016_G3_REPORTE WHERE
_TOP_LEVEL_AURI = "uuid:9e03e883-84bd-4457-a9cd-ae54746e1139" AND
_PARENT_AURI = "uuid:9e03e883-84bd-4457-a9cd-ae54746e1139"

you'll find 2 entries with matching _ORDINAL_NUMBER values (1).

Per the instructions (step 6 & 7), delete the older one of these.

On Mon, Jun 6, 2016 at 9:39 AM, Pablo Rodríguez < nuncaestardes...@gmail.com> wrote:

Well Now in another project I´m having the very same issue. wort
timing ever. In the middle of a big running project :S

There are not incomplete submissions according to ODK.

I was publishing a google spreadsheet and a fusion table (Running in
google app engine with billing enabled)

I´m looking at the log and I´m having an average of 4 to 7 error per
minute, basically like this:

18:14:55.688GET500956 B550 msAppEngine-Google; (+
http://code.google.com/appengine)/gae/uploadSubmissionsTask?fscUri=uuid%3A9672732c-1bc6-4dde-a463-d066b5f3edf3

0.1.0.2 - - [06/Jun/2016:18:14:55 +0200] "GET
/gae/uploadSubmissionsTask?fscUri=uuid%3A9672732c-1bc6-4dde-a463-d066b5f3edf3
HTTP/1.1" 500 956
http://olmedo-servicios-integrales.appspot.com/gae/watchdog
"AppEngine-Google; (+http://code.google.com/appengine)" "
olmedo-servicios-integrales.appspot.com" ms=550 cpu_ms=231
cpm_usd=2.5548e-7 loading_request=0
instance=00c61b117c327d751511c82e8ba427e6081789412cae8a6b2a33f690566e8b31
app_engine_release=1.9.38 trace_id=-

{

metadata: {…}

protoPayload: {…}

insertId: "2016-06-06|09:15:00.884913-07|10.106.162.86|1409825575"

log: "appengine.googleapis.com/request_log"

httpRequest: {…}

operation: {…} }

Seaching for OutOfMemory in the log and I don´t find anything.

Right now I just pulled all the information with Briefcase, and As
far as I can see I have more than what I had in my published tables.

My biggest concern is that I could loose submissions :S.

Any help is very appreciated.

Regards,

On Fri, May 6, 2016 at 6:22 PM, Mitch Sundt mitche...@gmail.com wrote:

This would indicate a data corruption issue.

See these instructions:
https://github.com/opendatakit/opendatakit/wiki/Aggregate-AppEngine-Troubleshooting#reparing-a-filled-in-form-submission

w.r.t. ODK Briefcase starting to pull data from the beginning every
time, if, after you pull data, you issue a push back up to the same server,
this will set tracking flags such that those records will be ignored and
skipped over the next time you pull.

ODK Briefcase stops at the first corrupted record because it is
fetching data in ascending marked-as-complete timestamp order. Until the
corrupted record is repaired or removed, it will not proceed to other
records.

Data corruption is more likely to occur if your server memory is
becoming exhausted or if you are submitting data over very-low-bandwidth
communications channels (e.g., satellite).

On Fri, May 6, 2016 at 5:52 AM, nuncaestardes...@gmail.com wrote:

El viernes, 6 de mayo de 2016, 12:30:39 (UTC+2), nuncaestardes...@gmail.com escribió:

El martes, 3 de mayo de 2016, 14:10:28 (UTC+2), nuncaestardes...@gmail.com escribió:

Hi guys,

I´m kind of new here, but I already have this problem:

On ODK Aggregate, when I want to export my data to .csv file,
It´s showing me an error (Failure - will retry later). it was working fine
but I guess the DDBB reach a certain volume that made it fail.

I have read that it is related to global max_allowed_packet
which has to be set to a higher value, with the following expression

set global max_allowed_packet = 1073741824;

I have my ODK aggregate deployed in a google app engine, so i
tied to type that in the cloud shell but nothing happened, I still have the
same problem, and as you can imaging it´s the worst moment to happend...

I have a excel macro that reads the csv file and download the
media files, renames and save everything in a file, so then i can open each
picture from a link in an excel spreadsheet.

Now I can only download the the CSV using ODK briefcase by:

1- Pulling all the data.

2- Exporting the CSV (I can only do it if I firstly pull the
data)

The problem is that by doing this I have to download the media
files twice (in briefcase and in my macro), and the CSV file exported by
briefcase contains the local link in my computer to the media files, but
not the cloud link to the media files, which in my case it is super
convenient. I figure out that the link to the media is a concatenation of
the ID number and some other things, so I manage to bypass this problem,
but I had to change my whole procedure in the middle of the work, making it
much more manual and not allowing me to use the tools I developped to
automate all this work.

Anyone knows how can I in ODK aggregate deployed in google app
engine execute this procedure?

set global max_allowed_packet = 1073741824;

IF that really is the problem, which i hope it is.

Thank you very much in advance.

Regards,

Pablo

One more thing, everytime I pull information with briefcase it
downloads the whole project instead of just downloading the new
entries,that could be something easy to improve in future versions, my
excel macro that downloads everything from the csv does that it is muc more
convenient this way.

Regards,

Pablo

I am trying to pull all the information with briefcase, and looking
at the log I have this:

...

fetching instance 3048 ...

fetching instance 3049 ...

fetching instance 3050 ...

fetching instance 3051 ...

fetching instance 3052 ...

fetching instance 3053 ...

retrieving next chunk of instances from server...

FAILED.

I don´t know if it failed because there is nothing more to pull or
because actually there is something elses happening... right now I don´t
know how many entries I have so 3053? I don´t know... the filter in
aggregate does not work so i don´t know how could I do this...

--

--

Post: opend...@googlegroups.com

Unsubscribe: opendatakit...@googlegroups.com

Options: http://groups.google.com/group/opendatakit?hl=en

You received this message because you are subscribed to the Google
Groups "ODK Community" group.

To unsubscribe from this group and stop receiving emails from it,
send an email to opendatakit...@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--

Mitch Sundt

Software Engineer

University of Washington

mitche...@gmail.com

--

--

Post: opend...@googlegroups.com

Unsubscribe: opendatakit...@googlegroups.com

Options: http://groups.google.com/group/opendatakit?hl=en

You received this message because you are subscribed to a topic in
the Google Groups "ODK Community" group.

To unsubscribe from this topic, visit
https://groups.google.com/d/topic/opendatakit/hjMKcHkk1tc/unsubscribe.

To unsubscribe from this group and all its topics, send an email to
opendatakit...@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--

--

Post: opend...@googlegroups.com

Unsubscribe: opendatakit...@googlegroups.com

Options: http://groups.google.com/group/opendatakit?hl=en

You received this message because you are subscribed to the Google
Groups "ODK Community" group.

To unsubscribe from this group and stop receiving emails from it,
send an email to opendatakit...@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--

Mitch Sundt

Software Engineer

University of Washington

mitche...@gmail.com

--

--

Post: opend...@googlegroups.com

Unsubscribe: opendatakit...@googlegroups.com

Options: http://groups.google.com/group/opendatakit?hl=en

You received this message because you are subscribed to a topic in the
Google Groups "ODK Community" group.

To unsubscribe from this topic, visit
https://groups.google.com/d/topic/opendatakit/hjMKcHkk1tc/unsubscribe.

To unsubscribe from this group and all its topics, send an email to
opendatakit...@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--

--

Post: opend...@googlegroups.com

Unsubscribe: opendatakit...@googlegroups.com

Options: http://groups.google.com/group/opendatakit?hl=en

You received this message because you are subscribed to the Google
Groups "ODK Community" group.

To unsubscribe from this group and stop receiving emails from it, send
an email to opendatakit...@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--

Mitch Sundt
Software Engineer
University of Washington
mitche...@gmail.com

--

Post: opendatakit@googlegroups.com
Unsubscribe: opendatakit+unsubscribe@googlegroups.com
Options: http://groups.google.com/group/opendatakit?hl=en

You received this message because you are subscribed to the Google Groups
"ODK Community" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to opendatakit+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--
Mitch Sundt
Software Engineer
University of Washington
mitchellsundt@gmail.com

Krishna_Rajesh · January 9, 2017, 5:32am

Congrats!

I'm glad you worked through the (very-confusing) sequence of steps and were able to get your system up and running again.

(It's been years since I worked on the PAUSED mechanism -- I was clearly mistaken about how it functions. I'm glad that it automatically resumed.)

I´ll complete my answer for the future:

Publishing is restored in both fusion tables and google spreadsheet (significantly slower in the spreadsheet), and now both status is "ACTIVE" (there is no button Mitch)

Briefcase gives me a "SUCCESS" message after pulling the whole thing.

CSV generation works without any problem.

Really this is the best!

IT WORKED!!!!

thank you very much!!!!!

the publishing started automatically, I guess that the 5-7 errors I was having per second was due to trying to publish.

I am extremely happy, not only for resolving this issue but for understanding it. Now I think I could do it again

Mitch you are the best!

You are very close. Note that in step (5), the instructions say:

Now, for each data table name and XXXXX value in step (2), execute a query of the form:SELECT * FROM opendatakit.YOUR_TABLE_NAME WHERE _PARENT_AURI = "XXXXXX"

Take special note of the backquote around the opendatakit.-qualified table name, and the double-quotes around the XXXXXX value. If the original query specified _DOM_AURI instead of_PARENT_AURI, use that. If this query does not return a value, you may also need to add the AND _TOP_LEVEL_AURI = "YYYYY" qualifier (but that should not be necessary), if it was present in the original error message.

In the queries you tried, you missed this part of the instructions (replace MAPFRE_JUNIO2016_G2_FOTO1_BN in your select statements with opendatakit.MAPFRE_JUNIO2016_G2_FOTO1_BN )

These instructions are saying to select and show to you the entries in the data tables that correspond to a specific submission (the one with instanceID uuid:9e03e883-84bd-4457-a9cd-ae54746e1139 ).

========================

If the publisher is paused, I believe the "PAUSED" is actually a button, and you can click on that to get it to resume.

That, of course, will only be successful if you fix the data corruption.

One last thing and hoping I can fix the main problem, since this is secondary: I tried to publish another spreadsheet (didnt´t work) and another fusion table (just existing submissions, 1420, when I had already published 1476 in the original) again, but at the end I decided to delete them. The two original spreadsheet and fusion table are both "BOTH Upload Existing & Stream New Submission Data" but now the status shows "PAUSED". How can I make them active again?

Ok, I think I´m seeing some progress in the sense of figuring out, what´s happening. I see where did you took the query you gave me, from Logs, App Engine, Background, All versions:

If I enlarge the first error I can see what was happening:

And I followed your instructions. But there is a second error, and it is different:

where it says:

org.opendatakit.aggregate.task.CsvWorkerImpl failureRecovery: Exception caught: org.opendatakit.common.datamodel.ODKEnumeratedElementException: Attachment errors:SELECT * FROM MAPFRE_JUNIO2016_G2_FOTO1_BN WHERE _TOP_LEVEL_AURI = uuid:9e03e883-84bd-4457-a9cd-ae54746e1139 AND _PARENT_AURI = uuid:9e03e883-84bd-4457-a9cd-ae54746e1139 is missing an attachment instance OR has extra copies. for MapfreJunio2016

(BTW, in MAPFRE_JUNIO2016_G2_FOTO1_BN G2 stands for group1 since in this form there are several fields displayed in the same screen, and FOTO1 means "picture1" which is a required field... I don´t know if that is relevant)

Then I decided to query:

SELECT * FROM MAPFRE_JUNIO2016_G2_FOTO1_BN WHERE _TOP_LEVEL_AURI = "uuid:9e03e883-84bd-4457-a9cd-ae54746e1139" AND _PARENT_AURI = "uuid:9e03e883-84bd-4457-a9cd-ae54746e1139"

But in this case instead of finding 2 entities there is no match. Also I can see that by searching this folder in the instances folder created by ODK Briefcase, instead of having an XML file and a picture/s it is empty.

So I guess now I should find a way of deleting uuid:9e03e883-84bd-4457-a9cd-ae54746e1139 in the database since it is a void submission, but I don´t know how to do this. ¿how can I do this?

What about all the rest of errors that I am finding in APP Engine,Default Service that I posted before? should I worry about them? (Because I do)

Why is this happening? How can I prevent this to happen again? could large pictures be a problem? (right now the pulled process made by briefcase is 4,24GB which includes XML files and pictures. This is 65% of what I am expecting)

Thank you very much in advance.

Ok So I see there are two errors constantly happening:

one of them:

09:55:58.212GET500960 B5.9 sAppEngine-Google; (+http://code.google.com/appengine)/gae/uploadSubmissionsTask?fscUri=uuid%3A9672732c-1bc6-4dde-a463-d066b5f3edf3

0.1.0.2 - - [07/Jun/2016:09:55:58 +0200] "GET /gae/uploadSubmissionsTask?fscUri=uuid%3A9672732c-1bc6-4dde-a463-d066b5f3edf3 HTTP/1.1" 500 960 http://olmedo-servicios-integrales.appspot.com/gae/watchdog "AppEngine-Google; (+http://code.google.com/appengine)" "olmedo-servicios-integrales.appspot.com" ms=5901 cpu_ms=460 cpm_usd=3.1940599999999997e-7 loading_request=0 instance=00c61b117c0e9e3d18748ca60cc00676c90a86e4344dfcefb7184ce68bcd8de904 app_engine_release=1.9.38 trace_id=-

{

metadata: {…}

protoPayload: {…}

insertId: "2016-06-07|00:56:08.408633-07|10.94.34.134|-1592936713"

log: "appengine.googleapis.com/request_log"

httpRequest: {…}

operation: {…} }

09:56:01.221org.opendatakit.aggregate.task.gae.servlet.UploadSubmissionsTaskServlet doGet: Beginning servlet processing

09:56:01.227org.opendatakit.aggregate.task.gae.servlet.UploadSubmissionsTaskServlet doGet: Request is running on frontend

09:56:01.228org.opendatakit.aggregate.task.UploadSubmissionsWorkerImpl uploadAllSubmissions: Beginning UPLOAD service: uuid:8979e040-b496-49c1-b8de-f355d69d4c46 form MapfreJunio2016

09:56:01.228org.opendatakit.aggregate.form.FormFactory internalGetForms: FormCache: using cached list of Forms

09:56:02.449org.opendatakit.aggregate.form.FormFactory internalGetForms: FormCache: using cached list of Forms

09:56:02.478[s~olmedo-servicios-integrales/1.392851787586356529].: Time left on lock: -73233127

09:56:02.478[s~olmedo-servicios-integrales/1.392851787586356529].: Time left on lock: -13406599

09:56:02.478[s~olmedo-servicios-integrales/1.392851787586356529].: Time left on lock: -52076838

09:56:02.478[s~olmedo-servicios-integrales/1.392851787586356529].: Time left on lock: -57872843

09:56:02.478[s~olmedo-servicios-integrales/1.392851787586356529].: Time left on lock: -78707824

And the other one

09:56:03.646GET2020 B1.4 sAppEngine-Google; (+http://code.google.com/appengine)/gae/uploadSubmissionsTask?fscUri=uuid%3A7ad28143-2a7a-4624-8807-215f05fd0102

0.1.0.2 - - [07/Jun/2016:09:56:03 +0200] "GET /gae/uploadSubmissionsTask?fscUri=uuid%3A7ad28143-2a7a-4624-8807-215f05fd0102 HTTP/1.1" 202 - http://olmedo-servicios-integrales.appspot.com/gae/watchdog "AppEngine-Google; (+http://code.google.com/appengine)" "olmedo-servicios-integrales.appspot.com" ms=1400 cpu_ms=120 cpm_usd=1.39586e-7 loading_request=0 instance=00c61b117c0e9e3d18748ca60cc00676c90a86e4344dfcefb7184ce68bcd8de904 app_engine_release=1.9.38 trace_id=-

{

metadata: {…}

protoPayload: {…}

insertId: "2016-06-07|00:56:08.408751-07|10.94.34.134|-1592936713"

log: "appengine.googleapis.com/request_log"

httpRequest: {…}

operation: {…} }

09:56:03.652org.opendatakit.aggregate.task.gae.servlet.UploadSubmissionsTaskServlet doGet: Beginning servlet processing

09:56:03.658org.opendatakit.aggregate.task.gae.servlet.UploadSubmissionsTaskServlet doGet: Request is running on frontend

09:56:03.658org.opendatakit.aggregate.task.UploadSubmissionsWorkerImpl uploadAllSubmissions: Beginning UPLOAD service: uuid:28dbf50b-2fcf-4424-800f-9c90eb823a09 form MapfreJunio2016

09:56:03.658org.opendatakit.aggregate.form.FormFactory internalGetForms: FormCache: using cached list of Forms

09:56:03.856org.opendatakit.aggregate.form.FormFactory internalGetForms: FormCache: using cached list of Forms

09:56:03.881[s~olmedo-servicios-integrales/1.392851787586356529].: Time left on lock: -14278749

09:56:03.881[s~olmedo-servicios-integrales/1.392851787586356529].: Time left on lock: -17503047

09:56:03.881[s~olmedo-servicios-integrales/1.392851787586356529].: Time left on lock: -34874322

09:56:03.881[s~olmedo-servicios-integrales/1.392851787586356529].: Time left on lock: -50316469

09:56:03.881[s~olmedo-servicios-integrales/1.392851787586356529].: Time left on lock: -18963331

So I try to run your query but changing the uuid to see if i can see two entries like the first thing I did but it says no entities matched. I checked and the publishing is still not working, but when I pull with Briefcase I can see it is downloading more instances, so I guess the system keeps receiving forms...

I´ve done exactly what do told me, now I´m pulling with briefcase to see if it works, but ends with FAILED

The problem is that I´m struggling to understand what I am doing, so the steps for Repairing a Filled-in Form Submission are not straight forward for me. I´ll keep trying...

I´m going to try Mitch, for me this is dancing on thin ice... let´s see...!

Thanks!

Have you worked through the database repair steps described here:

https://github.com/opendatakit/opendatakit/wiki/Aggregate-AppEngine-Troubleshooting#repairing-a-filled-in-form-submission

I.e.,

org.opendatakit.aggregate.task.gae.servlet.UploadSubmissionsTaskServlet doGet: org.opendatakit.aggregate.exception.ODKExternalServiceException: org.opendatakit.common.datamodel.ODKEnumeratedElementException: SELECT * FROM MAPFRE_JUNIO2016_G3_REPORTE WHERE _TOP_LEVEL_AURI = uuid:9e03e883-84bd-4457-a9cd-ae54746e1139 AND _PARENT_AURI = uuid:9e03e883-84bd-4457-a9cd-ae54746e1139 multiple-choice valueSet is missing an entry OR has an extra copy of one

and then, on the datastore tab,

entering in the query box:

SELECT * FROM opendatakit.MAPFRE_JUNIO2016_G3_REPORTE WHERE _TOP_LEVEL_AURI = "uuid:9e03e883-84bd-4457-a9cd-ae54746e1139" AND _PARENT_AURI = "uuid:9e03e883-84bd-4457-a9cd-ae54746e1139"

you'll find 2 entries with matching _ORDINAL_NUMBER values (1).

Per the instructions (step 6 & 7), delete the older one of these.

Well Now in another project I´m having the very same issue. wort timing ever. In the middle of a big running project :S

There are not incomplete submissions according to ODK.

I was publishing a google spreadsheet and a fusion table (Running in google app engine with billing enabled)

I´m looking at the log and I´m having an average of 4 to 7 error per minute, basically like this:

18:14:55.688GET500956 B550 msAppEngine-Google; (+http://code.google.com/appengine)/gae/uploadSubmissionsTask?fscUri=uuid%3A9672732c-1bc6-4dde-a463-d066b5f3edf3

0.1.0.2 - - [06/Jun/2016:18:14:55 +0200] "GET /gae/uploadSubmissionsTask?fscUri=uuid%3A9672732c-1bc6-4dde-a463-d066b5f3edf3 HTTP/1.1" 500 956 http://olmedo-servicios-integrales.appspot.com/gae/watchdog "AppEngine-Google; (+http://code.google.com/appengine)" "olmedo-servicios-integrales.appspot.com" ms=550 cpu_ms=231 cpm_usd=2.5548e-7 loading_request=0 instance=00c61b117c327d751511c82e8ba427e6081789412cae8a6b2a33f690566e8b31 app_engine_release=1.9.38 trace_id=-

{

metadata: {…}

protoPayload: {…}

insertId: "2016-06-06|09:15:00.884913-07|10.106.162.86|1409825575"

log: "appengine.googleapis.com/request_log"

httpRequest: {…}

operation: {…} }

Seaching for OutOfMemory in the log and I don´t find anything.

Right now I just pulled all the information with Briefcase, and As far as I can see I have more than what I had in my published tables.

My biggest concern is that I could loose submissions :S.

Any help is very appreciated.

Regards,

This would indicate a data corruption issue.

See these instructions: https://github.com/opendatakit/opendatakit/wiki/Aggregate-AppEngine-Troubleshooting#reparing-a-filled-in-form-submission

w.r.t. ODK Briefcase starting to pull data from the beginning every time, if, after you pull data, you issue a push back up to the same server, this will set tracking flags such that those records will be ignored and skipped over the next time you pull.

ODK Briefcase stops at the first corrupted record because it is fetching data in ascending marked-as-complete timestamp order. Until the corrupted record is repaired or removed, it will not proceed to other records.

Data corruption is more likely to occur if your server memory is becoming exhausted or if you are submitting data over very-low-bandwidth communications channels (e.g., satellite).

Hi guys,

I´m kind of new here, but I already have this problem:

On ODK Aggregate, when I want to export my data to .csv file, It´s showing me an error (Failure - will retry later). it was working fine but I guess the DDBB reach a certain volume that made it fail.

I have read that it is related to global max_allowed_packet which has to be set to a higher value, with the following expression

set global max_allowed_packet = 1073741824;

I have my ODK aggregate deployed in a google app engine, so i tied to type that in the cloud shell but nothing happened, I still have the same problem, and as you can imaging it´s the worst moment to happend...

I have a excel macro that reads the csv file and download the media files, renames and save everything in a file, so then i can open each picture from a link in an excel spreadsheet.

Now I can only download the the CSV using ODK briefcase by:

1- Pulling all the data.

2- Exporting the CSV (I can only do it if I firstly pull the data)

The problem is that by doing this I have to download the media files twice (in briefcase and in my macro), and the CSV file exported by briefcase contains the local link in my computer to the media files, but not the cloud link to the media files, which in my case it is super convenient. I figure out that the link to the media is a concatenation of the ID number and some other things, so I manage to bypass this problem, but I had to change my whole procedure in the middle of the work, making it much more manual and not allowing me to use the tools I developped to automate all this work.

Anyone knows how can I in ODK aggregate deployed in google app engine execute this procedure?

set global max_allowed_packet = 1073741824;

IF that really is the problem, which i hope it is.

Thank you very much in advance.

Regards,

Pablo

One more thing, everytime I pull information with briefcase it downloads the whole project instead of just downloading the new entries,that could be something easy to improve in future versions, my excel macro that downloads everything from the csv does that it is muc more convenient this way.

Regards,

Pablo

I am trying to pull all the information with briefcase, and looking at the log I have this:

...

fetching instance 3048 ...

fetching instance 3049 ...

fetching instance 3050 ...

fetching instance 3051 ...

fetching instance 3052 ...

fetching instance 3053 ...

retrieving next chunk of instances from server...

FAILED.

I don´t know if it failed because there is nothing more to pull or because actually there is something elses happening... right now I don´t know how many entries I have so 3053? I don´t know... the filter in aggregate does not work so i don´t know how could I do this...

--

--

Post: opend...@googlegroups.com

Unsubscribe: opendatakit...@googlegroups.com

Options: http://groups.google.com/group/opendatakit?hl=en

You received this message because you are subscribed to the Google Groups "ODK Community" group.

To unsubscribe from this group and stop receiving emails from it, send an email to opendatakit...@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--

Mitch Sundt

Software Engineer

University of Washington

mitche...@gmail.com

--

--

Post: opend...@googlegroups.com

Unsubscribe: opendatakit...@googlegroups.com

Options: http://groups.google.com/group/opendatakit?hl=en

You received this message because you are subscribed to a topic in the Google Groups "ODK Community" group.

To unsubscribe from this topic, visit https://groups.google.com/d/topic/opendatakit/hjMKcHkk1tc/unsubscribe.

To unsubscribe from this group and all its topics, send an email to opendatakit...@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--

--

Post: opend...@googlegroups.com

Unsubscribe: opendatakit...@googlegroups.com

Options: http://groups.google.com/group/opendatakit?hl=en

You received this message because you are subscribed to the Google Groups "ODK Community" group.

To unsubscribe from this group and stop receiving emails from it, send an email to opendatakit...@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--

Mitch Sundt

Software Engineer

University of Washington

mitche...@gmail.com

--

--

Post: opend...@googlegroups.com

Unsubscribe: opendatakit...@googlegroups.com

Options: http://groups.google.com/group/opendatakit?hl=en

You received this message because you are subscribed to a topic in the Google Groups "ODK Community" group.

To unsubscribe from this topic, visit https://groups.google.com/d/topic/opendatakit/hjMKcHkk1tc/unsubscribe.

To unsubscribe from this group and all its topics, send an email to opendatakit...@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--

--

Post: opend...@googlegroups.com

Unsubscribe: opendatakit...@googlegroups.com

Options: http://groups.google.com/group/opendatakit?hl=en

You received this message because you are subscribed to the Google Groups "ODK Community" group.

To unsubscribe from this group and stop receiving emails from it, send an email to opendatakit...@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--

Mitch Sundt

Software Engineer

University of Washington

mitche...@gmail.com

--

--

Post: opend...@googlegroups.com

Unsubscribe: opendatakit...@googlegroups.com

Options: http://groups.google.com/group/opendatakit?hl=en

You received this message because you are subscribed to the Google Groups "ODK Community" group.

To unsubscribe from this group and stop receiving emails from it, send an email to opendatakit...@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--

Mitch Sundt
Software Engineer
University of Washington
mitche...@gmail.com

Hi Guys,

I am trying to export JSON format. The last downloaded file size is 43.7 MB. Now when I try to export, it is showing 'Generation in progress'. Nothing happens. When I checked the log (catalina.out) I am getting the following:

/var/log/tomcat7# tail -f catalina.out
Jan 09, 2017 10:45:02 AM org.opendatakit.aggregate.task.CsvWorkerImpl generateJsonFile
INFO: wrapping up JSON generation for NCD_DMS_8Nov2016_v26
Jan 09, 2017 10:45:02 AM org.opendatakit.aggregate.task.CsvWorkerImpl generateJsonFile
WARNING: stale JSON activity - do not save file in PersistentResults table for NCD_DMS_8Nov2016_v26
Jan 09, 2017 10:45:02 AM org.opendatakit.aggregate.task.CsvWorkerImpl generateJsonFile
INFO: iteration 46 before issuing query for NCD_DMS_8Nov2016_v26
Jan 09, 2017 10:45:02 AM org.opendatakit.aggregate.task.CsvWorkerImpl generateJsonFile
INFO: wrapping up JSON generation for NCD_DMS_8Nov2016_v26
Jan 09, 2017 10:45:02 AM org.opendatakit.aggregate.task.CsvWorkerImpl generateJsonFile
WARNING: stale JSON activity - do not save file in PersistentResults table for NCD_DMS_8Nov2016_v26

Please help me. This is a live project and all are waiting for this output to proceed.

Thanks in advance
Krishna

···

On Wednesday, June 8, 2016 at 10:57:43 PM UTC+5:30, Mitch Sundt wrote: > On Wed, Jun 8, 2016 at 1:23 AM, wrote: > El miércoles, 8 de junio de 2016, 9:30:55 (UTC+2), tomadedat...@gmail.com escribió: > > El martes, 7 de junio de 2016, 23:26:31 (UTC+2), Mitch Sundt escribió: > > > On Tue, Jun 7, 2016 at 10:05 AM, Pablo Rodríguez wrote: > > > On Tue, Jun 7, 2016 at 5:32 PM, Pablo Rodríguez wrote: > > > On Tue, Jun 7, 2016 at 10:09 AM, Pablo Rodríguez wrote: > > > On Tue, Jun 7, 2016 at 9:51 AM, Pablo Rodríguez wrote: > > > On Tue, Jun 7, 2016 at 9:22 AM, wrote: > > > El martes, 7 de junio de 2016, 0:55:47 (UTC+2), Mitch Sundt escribió: > > > > On Mon, Jun 6, 2016 at 9:39 AM, Pablo Rodríguez wrote: > > > > On Fri, May 6, 2016 at 6:22 PM, Mitch Sundt wrote: > > > > On Fri, May 6, 2016 at 5:52 AM, wrote: > > > > El viernes, 6 de mayo de 2016, 12:30:39 (UTC+2), nuncaestardes...@gmail.com escribió: > > > > > El martes, 3 de mayo de 2016, 14:10:28 (UTC+2), nuncaestardes...@gmail.com escribió: