Unknown Error: "Error: Problem persisting data or accessing data

Hi all,

We have a survey going live at the moment, with data being uploaded on a
daily basis. We are using a server a google app engine deployment of ODK
Aggregate version 1.4.3. It was my intention to update this to the latest
version of Aggregate once this fieldwork was completed.

To date, there has been no issues, but starting today when viewing data on
the "Submission" page of our appspot page for this server, I have been
getting the below error. The error is repetitive, but each time the uuid
case number changes.

"Error: Problem persisting data or accessing data
([ChoiceSubmissionType.getValueFromEntity] SQL: SELECT * FROM
opendatakit.ASUBL_FINAL2_ELG_B_SEC2_MJ_G6_J10C WHERE _PARENT_AURI =
uuid:6219f226-5a47-4501-b439-061dfce52d34 ORDER BY _PARENT_AURI exception:
This
request (2d170ed258bfabee) started at 2015/03/20 16:56:04.949 UTC and was
still
executing at 2015/03/20 16:57:04.779 UTC.)"

In the app engine application, when looking at the log, there are a number
of Info messages but thy all vary, and one warning message (see below):

  1. W2015-03-20 10:17:58.727 A problem was encountered with the process
    that handled this request, causing it to exit. This is likely to cause a
    new process to be used for the nex

Any advice on how to address this error, and if it will cause a potential
loss of data (all data has been backed up) in the future?

Thanks,

~Lloyd

Several things could be going on:

(1) if many users are all watching the ODK Aggregate webpages at the same
time, they may be competing for processing space and time. Restrict access
to the server and provide nightly reports or publish the data into Google
Fusion Tables and provide access to that.

(2) if you are using "Export to CSV" to produce reports, switch to using
ODK Briefcase to download data to your local computer and then export to
CSV from there. The Export functionality on ODK Aggregate holds all of the
dataset in memory when generating a report. This will not work beyond a
certain size dataset.

(3) increase the "Frontend Instance Class" (machine size) on the AppEngine
( https://appengine.google.com/ ). Choose your application id, then click
on Application Settings on the left sidebar. And select a larger Frontend
Instance Class.

Note that increasing the machine size will increase your costs.

··· On Fri, Mar 20, 2015 at 10:31 AM, Lloyd Owen Banwart < lloyd.banwart@gmail.com> wrote:

Hi all,

We have a survey going live at the moment, with data being uploaded on a
daily basis. We are using a server a google app engine deployment of ODK
Aggregate version 1.4.3. It was my intention to update this to the latest
version of Aggregate once this fieldwork was completed.

To date, there has been no issues, but starting today when viewing data on
the "Submission" page of our appspot page for this server, I have been
getting the below error. The error is repetitive, but each time the uuid
case number changes.

"Error: Problem persisting data or accessing data
([ChoiceSubmissionType.getValueFromEntity] SQL: SELECT * FROM
opendatakit.ASUBL_FINAL2_ELG_B_SEC2_MJ_G6_J10C WHERE _PARENT_AURI =
uuid:6219f226-5a47-4501-b439-061dfce52d34 ORDER BY _PARENT_AURI exception:
This
request (2d170ed258bfabee) started at 2015/03/20 16:56:04.949 UTC and was
still
executing at 2015/03/20 16:57:04.779 UTC.)"

In the app engine application, when looking at the log, there are a number
of Info messages but thy all vary, and one warning message (see below):

  1. W2015-03-20 10:17:58.727 A problem was encountered with the process
    that handled this request, causing it to exit. This is likely to cause a
    new process to be used for the nex

Any advice on how to address this error, and if it will cause a potential
loss of data (all data has been backed up) in the future?

Thanks,

~Lloyd

--

Post: opendatakit@googlegroups.com
Unsubscribe: opendatakit+unsubscribe@googlegroups.com
Options: http://groups.google.com/group/opendatakit?hl=en


You received this message because you are subscribed to the Google Groups
"ODK Community" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to opendatakit+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--
Mitch Sundt
Software Engineer
University of Washington
mitchellsundt@gmail.com

I logged in to my Aggregate Account but I am unable to filter submission
for a particualar form. These are the errors

Error: Problem persisting data or accessing data
([ChoiceSubmissionType.getValueFromEntity] SQL: SELECT * FROM
opendatakit.BRSTDY_MAIN_BLOCK_D_D10_D10_20 WHERE _PARENT_AURI =
uuid:1b9a6758-a912-47d7-af3a-2a9c38fa030f ORDER BY _PARENT_AURI exception:
This request (42957662ed0ed296) started at 2016/08/13 05:13:27.621 UTC and
was still executing at 2016/08/13 05:14:27.525 UTC.)

I have gone through the discussions on the Google group and followed this
link


but it seems am not getting it. I will appreciate any help from you on how
to fix this.
··· On Friday, March 20, 2015 at 7:31:28 PM UTC+2, Lloyd Owen Banwart wrote: > > Hi all, > > We have a survey going live at the moment, with data being uploaded on a > daily basis. We are using a server a google app engine deployment of ODK > Aggregate version 1.4.3. It was my intention to update this to the latest > version of Aggregate once this fieldwork was completed. > > To date, there has been no issues, but starting today when viewing data on > the "Submission" page of our appspot page for this server, I have been > getting the below error. The error is repetitive, but each time the uuid > case number changes. > > "Error: Problem persisting data or accessing data > ([ChoiceSubmissionType.getValueFromEntity] SQL: SELECT * FROM > opendatakit.ASUBL_FINAL2_ELG_B_SEC2_MJ_G6_J10C WHERE _PARENT_AURI = > uuid:6219f226-5a47-4501-b439-061dfce52d34 ORDER BY _PARENT_AURI exception: > This > request (2d170ed258bfabee) started at 2015/03/20 16:56:04.949 UTC and was > still > executing at 2015/03/20 16:57:04.779 UTC.)" > > > In the app engine application, when looking at the log, there are a number > of Info messages but thy all vary, and one warning message (see below): > > > 1. W2015-03-20 10:17:58.727 A problem was encountered with the process > that handled this request, causing it to exit. This is likely to cause a > new process to be used for the nex > > > Any advice on how to address this error, and if it will cause a potential > loss of data (all data has been backed up) in the future? > > Thanks, > > ~Lloyd > >

Hi Everyone,

I have the following error when I try to upload my forms into the ODK Aggregate:
Error: Problem persisting data or accessing data (org.springframework.dao.TransientDataAccessResourceException: PreparedStatementCallback; SQL [SELECT _URI, _CREATOR_URI_USER, _CREATION_DATE, _LAST_UPDATE_URI_USER, _LAST_UPDATE_DATE, _MODEL_VERSION, _UI_VERSION, _IS_COMPLETE, _SUBMISSION_DATE, _MARKED_AS_COMPLETE_DATE, GRP2_GRP2A_HIVMEDPREG, HOSPITAL, MOMFNM, GRP2_GRP2A_GRP2B_P1_FREQ, OTH_TRIBE, DEVICEID, GRP1_OTHERNAT, GRP2_LSTTEST, REPORTDATE, GRP2_RESULT, GRP2_GRP2A_GRP2B_OTHER_P3, GRP2_GRP2A_GRP2B_OTHER_P2, GRP1_MOMPHONE, GRP1_OTHERWATER, META_INSTANCE_ID, GRP1_MOMHEIGHT, GRP1_WATER, TRIBE, MOMSNM, GRP1_MOMWEIGHT2, GRP1_MOMWEIGHT1, GRP1_SMOKERS, GRP1_NATIONALITY, GRP2_GRP2A_GRP2B_OTHER_P1, GRP2_GRP2A_GRP2B_OTHER_B1, GRP1_CURLIVE, SIGNED_CONSENT, GRP2_FIRSTTEST, ID_ANCIDNO, GRP1_JOB, GRP1_CHLDNO, GRP1_JOBTYPE, TODAY, GRP1_INCOME, PAPFNM, GRP1_OTHERLIVE, GRP1_MOMMARST, GRP2_GRP2A_GRP2B_P3_FREQ, GRP2_GRP2A_GRP2B_P2_FREQ, CONSENT, PAPSNM, INFO8, INFO7, INFO4, GRP1_SCHLEV, INFO1, GRP2_GRP2A_GRP2B_WHYSMISS, SURVEY_COMPLETED, ID_OTHIDNO, GRP1_OTHERTOILET, GRP2_GRP2A_TRADHLR, EVERTEST, GRP2_GRP2A_GRP2B_MISSHIVMED, GRP1_RMNO, GRP2_GRP2A_GRP2B_B1_FREQ, ID_IPIDNO, ID_ARTIDNO, QCINITIAL1, SURVEY_DECLINE, VILLAGE, GRP1_HSEPPLNO, START, ID_PMTCTIDNO, ID_UNQIDNO, END, GRP2_GRP2A_RESULT2, GRP1_TOILET FROM odk_birthdefects.BIRTH_DEFECTS_CORE WHERE _MARKED_AS_COMPLETE_DATE > ? AND _IS_COMPLETE = ? ORDER BY _MARKED_AS_COMPLETE_DATE ASC , _URI ASC ;]; Packet for query is too large (1508 > 1024). You can change this value on the server by setting the max_allowed_packet' variable.; nested exception is com.mysql.jdbc.PacketTooBigException: Packet for query is too large (1508 > 1024). You can change this value on the server by setting the max_allowed_packet' variable.)

Any thoughts on why I am having this problem and how I can work it out?

Vincent

Mitch. Thanks (as always) for the clear and timely feedback.

  1. I've limited access, but it was a total of 3 with access before (now it
    is a total of 2) to the server.

  2. We use ODK briefcase (as we avail of the Stata ODKmeta program to
    provide value/var labels and restructure data sets from repeat groups).

  3. Re: Increasing the machine size; this is one of the smaller data
    collections we have done. The survey is large, but we are collecting data
    from less than 2,000 households --- is there a reason this issue would
    arise now and it hasn't in the past? Regardless, I'm waiting to see if
    limiting access to the server addresses this issue, if it does not I will
    increase the machine size.

I'll post progress here in the event someone else faces this issue in the
future. Thanks again.

~Lloyd

··· On Friday, March 20, 2015 at 10:35:18 PM UTC-4, Mitch Sundt wrote: > > Several things could be going on: > > (1) if many users are all watching the ODK Aggregate webpages at the same > time, they may be competing for processing space and time. Restrict access > to the server and provide nightly reports or publish the data into Google > Fusion Tables and provide access to that. > > (2) if you are using "Export to CSV" to produce reports, switch to using > ODK Briefcase to download data to your local computer and then export to > CSV from there. The Export functionality on ODK Aggregate holds all of the > dataset in memory when generating a report. This will not work beyond a > certain size dataset. > > (3) increase the "Frontend Instance Class" (machine size) on the AppEngine > ( https://appengine.google.com/ ). Choose your application id, then click > on Application Settings on the left sidebar. And select a larger Frontend > Instance Class. > > Note that increasing the machine size will increase your costs. > > > On Fri, Mar 20, 2015 at 10:31 AM, Lloyd Owen Banwart <lloyd....@gmail.com > wrote: > >> Hi all, >> >> We have a survey going live at the moment, with data being uploaded on a >> daily basis. We are using a server a google app engine deployment of ODK >> Aggregate version 1.4.3. It was my intention to update this to the latest >> version of Aggregate once this fieldwork was completed. >> >> To date, there has been no issues, but starting today when viewing data >> on the "Submission" page of our appspot page for this server, I have been >> getting the below error. The error is repetitive, but each time the uuid >> case number changes. >> >> "Error: Problem persisting data or accessing data >> ([ChoiceSubmissionType.getValueFromEntity] SQL: SELECT * FROM >> opendatakit.ASUBL_FINAL2_ELG_B_SEC2_MJ_G6_J10C WHERE _PARENT_AURI = >> uuid:6219f226-5a47-4501-b439-061dfce52d34 ORDER BY _PARENT_AURI >> exception: This >> request (2d170ed258bfabee) started at 2015/03/20 16:56:04.949 UTC and was >> still >> executing at 2015/03/20 16:57:04.779 UTC.)" >> >> >> In the app engine application, when looking at the log, there are a >> number of Info messages but thy all vary, and one warning message (see >> below): >> >> >> 1. W2015-03-20 10:17:58.727 A problem was encountered with the >> process that handled this request, causing it to exit. This is likely to >> cause a new process to be used for the nex >> >> >> Any advice on how to address this error, and if it will cause a potential >> loss of data (all data has been backed up) in the future? >> >> Thanks, >> >> ~Lloyd >> >> -- >> -- >> Post: opend...@googlegroups.com >> Unsubscribe: opendatakit...@googlegroups.com >> Options: http://groups.google.com/group/opendatakit?hl=en >> >> --- >> You received this message because you are subscribed to the Google Groups >> "ODK Community" group. >> To unsubscribe from this group and stop receiving emails from it, send an >> email to opendatakit...@googlegroups.com . >> For more options, visit https://groups.google.com/d/optout. >> > > > > -- > Mitch Sundt > Software Engineer > University of Washington > mitche...@gmail.com >

Mitch,

I have a problem with my aggregate. I am collecting data and trying to post
it using the ODK collect but it is not being transmitted. Secondly when
trying to view the collected data from the aggregate, i can scroll through
the first two pages but on going to the next page i get this

Error: Problem persisting data or accessing data (SELECT * FROM
BUILD_284588833_BASIC_INFORMATION_CURENT_WATER_USE WHERE _TOP_LEVEL_AURI =
uuid:b3358b24-8813-49bc-917e-a7d82801fe14 AND _PARENT_AURI =
uuid:b3358b24-8813-49bc-917e-a7d82801fe14 multiple-choice valueSet is
missing an entry OR has an extra copy of one)

When i query it by GQL

SELECT * FROM BUILD_284588833_BASIC_INFORMATION_CURENT_WATER_USE WHERE
_TOP_LEVEL_AURI = "uuid:b3358b24-8813-49bc-917e-a7d82801fe14" AND
_PARENT_AURI = "uuid:b3358b24-8813-49bc-917e-a7d82801fe14"

i get No entities matched this query

What does this mean and how can i rectify it?

··· On Saturday, March 21, 2015 at 5:35:18 AM UTC+3, Mitch Sundt wrote: > > Several things could be going on: > > (1) if many users are all watching the ODK Aggregate webpages at the same > time, they may be competing for processing space and time. Restrict access > to the server and provide nightly reports or publish the data into Google > Fusion Tables and provide access to that. > > (2) if you are using "Export to CSV" to produce reports, switch to using > ODK Briefcase to download data to your local computer and then export to > CSV from there. The Export functionality on ODK Aggregate holds all of the > dataset in memory when generating a report. This will not work beyond a > certain size dataset. > > (3) increase the "Frontend Instance Class" (machine size) on the AppEngine > ( https://appengine.google.com/ ). Choose your application id, then click > on Application Settings on the left sidebar. And select a larger Frontend > Instance Class. > > Note that increasing the machine size will increase your costs. > > > On Fri, Mar 20, 2015 at 10:31 AM, Lloyd Owen Banwart <lloyd....@gmail.com > wrote: > >> Hi all, >> >> We have a survey going live at the moment, with data being uploaded on a >> daily basis. We are using a server a google app engine deployment of ODK >> Aggregate version 1.4.3. It was my intention to update this to the latest >> version of Aggregate once this fieldwork was completed. >> >> To date, there has been no issues, but starting today when viewing data >> on the "Submission" page of our appspot page for this server, I have been >> getting the below error. The error is repetitive, but each time the uuid >> case number changes. >> >> "Error: Problem persisting data or accessing data >> ([ChoiceSubmissionType.getValueFromEntity] SQL: SELECT * FROM >> opendatakit.ASUBL_FINAL2_ELG_B_SEC2_MJ_G6_J10C WHERE _PARENT_AURI = >> uuid:6219f226-5a47-4501-b439-061dfce52d34 ORDER BY _PARENT_AURI >> exception: This >> request (2d170ed258bfabee) started at 2015/03/20 16:56:04.949 UTC and was >> still >> executing at 2015/03/20 16:57:04.779 UTC.)" >> >> >> In the app engine application, when looking at the log, there are a >> number of Info messages but thy all vary, and one warning message (see >> below): >> >> >> 1. W2015-03-20 10:17:58.727 A problem was encountered with the >> process that handled this request, causing it to exit. This is likely to >> cause a new process to be used for the nex >> >> >> Any advice on how to address this error, and if it will cause a potential >> loss of data (all data has been backed up) in the future? >> >> Thanks, >> >> ~Lloyd >> >> -- >> -- >> Post: opend...@googlegroups.com >> Unsubscribe: opendatakit...@googlegroups.com >> Options: http://groups.google.com/group/opendatakit?hl=en >> >> --- >> You received this message because you are subscribed to the Google Groups >> "ODK Community" group. >> To unsubscribe from this group and stop receiving emails from it, send an >> email to opendatakit...@googlegroups.com . >> For more options, visit https://groups.google.com/d/optout. >> > > > > -- > Mitch Sundt > Software Engineer > University of Washington > mitche...@gmail.com >

Hi Vincent,

I believe I have encountered this once as well, but am struggling to
remember where the issue lay (apache/tomcat/mysql). My intuition is that it
is mysql. Check your my.cnf (usually located in /etc/mysql) for the
variable max_allowed_packet, and increase it. Mine is currently set to
16MB. And remember to restart mysql after you make the change...

··· On Thu, Oct 27, 2016 at 4:02 AM, wrote:

Hi Everyone,

I have the following error when I try to upload my forms into the ODK
Aggregate:
Error: Problem persisting data or accessing data (org.springframework.dao.
TransientDataAccessResourceException: PreparedStatementCallback; SQL
[SELECT _URI, _CREATOR_URI_USER, _CREATION_DATE,
_LAST_UPDATE_URI_USER, _LAST_UPDATE_DATE, _MODEL_VERSION,
_UI_VERSION, _IS_COMPLETE, _SUBMISSION_DATE,
_MARKED_AS_COMPLETE_DATE, GRP2_GRP2A_HIVMEDPREG, HOSPITAL, MOMFNM,
GRP2_GRP2A_GRP2B_P1_FREQ, OTH_TRIBE, DEVICEID, GRP1_OTHERNAT,
GRP2_LSTTEST, REPORTDATE, GRP2_RESULT, GRP2_GRP2A_GRP2B_OTHER_P3,
GRP2_GRP2A_GRP2B_OTHER_P2, GRP1_MOMPHONE, GRP1_OTHERWATER,
META_INSTANCE_ID, GRP1_MOMHEIGHT, GRP1_WATER, TRIBE, MOMSNM,
GRP1_MOMWEIGHT2, GRP1_MOMWEIGHT1, GRP1_SMOKERS, GRP1_NATIONALITY,
GRP2_GRP2A_GRP2B_OTHER_P1, GRP2_GRP2A_GRP2B_OTHER_B1, GRP1_CURLIVE,
SIGNED_CONSENT, GRP2_FIRSTTEST, ID_ANCIDNO, GRP1_JOB,
GRP1_CHLDNO, GRP1_JOBTYPE, TODAY, GRP1_INCOME, PAPFNM,
GRP1_OTHERLIVE, GRP1_MOMMARST, GRP2_GRP2A_GRP2B_P3_FREQ,
GRP2_GRP2A_GRP2B_P2_FREQ, CONSENT, PAPSNM, INFO8, INFO7, INFO4,
GRP1_SCHLEV, INFO1, GRP2_GRP2A_GRP2B_WHYSMISS, SURVEY_COMPLETED,
ID_OTHIDNO, GRP1_OTHERTOILET, GRP2_GRP2A_TRADHLR, EVERTEST,
GRP2_GRP2A_GRP2B_MISSHIVMED, GRP1_RMNO, GRP2_GRP2A_GRP2B_B1_FREQ,
ID_IPIDNO, ID_ARTIDNO, QCINITIAL1, SURVEY_DECLINE, VILLAGE,
GRP1_HSEPPLNO, START, ID_PMTCTIDNO, ID_UNQIDNO, END,
GRP2_GRP2A_RESULT2, GRP1_TOILET FROM odk_birthdefects.BIRTH_DEFECTS_CORE
WHERE _MARKED_AS_COMPLETE_DATE > ? AND _IS_COMPLETE = ? ORDER BY
_MARKED_AS_COMPLETE_DATE ASC , _URI ASC ;]; Packet for query is too
large (1508 > 1024). You can change this value on the server by setting the
max_allowed_packet' variable.; nested exception is com.mysql.jdbc.PacketTooBigException:
Packet for query is too large (1508 > 1024). You can change this value on
the server by setting the max_allowed_packet' variable.)

Any thoughts on why I am having this problem and how I can work it out?

Vincent

--

Post: opendatakit@googlegroups.com
Unsubscribe: opendatakit+unsubscribe@googlegroups.com
Options: http://groups.google.com/group/opendatakit?hl=en


You received this message because you are subscribed to a topic in the
Google Groups "ODK Community" group.
To unsubscribe from this topic, visit https://groups.google.com/d/
topic/opendatakit/gF24RHrltGs/unsubscribe.
To unsubscribe from this group and all its topics, send an email to
opendatakit+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--
Colin

To follow upon this post.

I had decreased access to the server for 24 hours to two users, and this
did not resolve the issue of receiving the below error. I then steadily
increased the frontend instance Class from 256 to 512 to 1024. In each
case, the error still was displayed on the ODK server. This error only
displays when viewing the data on the "Submissions tab" and under "Filter
Submissions" (no filter is applied). The error is repetitive, meaning that
when I leave the browser open, the error will display, instances may (or
may not) be shown in the submissions tab, then the error will pop up again.
Each time the error displays the error appears identical other than the
uuid, specific Q it is referencing, and start and end time of request.
I've tracked most uuids that have shown in the errors, and cross
referenced this with data obtained after the error has been displayed. All
instances displayed during errors are in the data which have been
downloaded, and the instances appear unaffected.

When accessing the data (via the "Export" function or via ODK Briefcase)
there appears to be no data loss when check against field-records of
uploaded instances compared to instances obtained from the server. Other
than being a annoying, and limiting any quick review of recently submitted
data via the server, I don't see any major issues at the moment with the
error -- but would love to get to the bottom of it.

Second example of error:
Error: Problem persisting data or accessing data
([ChoiceSubmissionType.getValueFromEntity] SQL: SELECT * FROM
opendatakit.ASUBL_FINAL2_ELG_B_SEC2_SEC3_MM_G3_LABELM3_2 WHERE _PARENT_AURI
= uuid:e1ba7512-9c8e-47d1-8458-0ddc1935642b ORDER BY _PARENT_AURI
exception: This request (f44144cff1ba1b4f) started at 2015/03/23
13:51:54.645 UTC and was still executing at 2015/03/23 13:52:54.505 UTC.)

··· On Monday, March 23, 2015 at 9:39:14 AM UTC-4, Lloyd Owen Banwart wrote: > > Mitch. Thanks (as always) for the clear and timely feedback. > > 1) I've limited access, but it was a total of 3 with access before (now it > is a total of 2) to the server. > > 2) We use ODK briefcase (as we avail of the Stata ODKmeta program to > provide value/var labels and restructure data sets from repeat groups). > > 3) Re: Increasing the machine size; this is one of the smaller data > collections we have done. The survey is large, but we are collecting data > from less than 2,000 households --- is there a reason this issue would > arise now and it hasn't in the past? Regardless, I'm waiting to see if > limiting access to the server addresses this issue, if it does not I will > increase the machine size. > > I'll post progress here in the event someone else faces this issue in the > future. Thanks again. > > ~Lloyd > > On Friday, March 20, 2015 at 10:35:18 PM UTC-4, Mitch Sundt wrote: >> >> Several things could be going on: >> >> (1) if many users are all watching the ODK Aggregate webpages at the same >> time, they may be competing for processing space and time. Restrict access >> to the server and provide nightly reports or publish the data into Google >> Fusion Tables and provide access to that. >> >> (2) if you are using "Export to CSV" to produce reports, switch to using >> ODK Briefcase to download data to your local computer and then export to >> CSV from there. The Export functionality on ODK Aggregate holds all of the >> dataset in memory when generating a report. This will not work beyond a >> certain size dataset. >> >> (3) increase the "Frontend Instance Class" (machine size) on the >> AppEngine ( https://appengine.google.com/ ). Choose your application id, >> then click on Application Settings on the left sidebar. And select a larger >> Frontend Instance Class. >> >> Note that increasing the machine size will increase your costs. >> >> >> On Fri, Mar 20, 2015 at 10:31 AM, Lloyd Owen Banwart > >>> Hi all, >>> >>> We have a survey going live at the moment, with data being uploaded on a >>> daily basis. We are using a server a google app engine deployment of ODK >>> Aggregate version 1.4.3. It was my intention to update this to the latest >>> version of Aggregate once this fieldwork was completed. >>> >>> To date, there has been no issues, but starting today when viewing data >>> on the "Submission" page of our appspot page for this server, I have been >>> getting the below error. The error is repetitive, but each time the uuid >>> case number changes. >>> >>> "Error: Problem persisting data or accessing data >>> ([ChoiceSubmissionType.getValueFromEntity] SQL: SELECT * FROM >>> opendatakit.ASUBL_FINAL2_ELG_B_SEC2_MJ_G6_J10C WHERE _PARENT_AURI = >>> uuid:6219f226-5a47-4501-b439-061dfce52d34 ORDER BY _PARENT_AURI >>> exception: This >>> request (2d170ed258bfabee) started at 2015/03/20 16:56:04.949 UTC and >>> was still >>> executing at 2015/03/20 16:57:04.779 UTC.)" >>> >>> >>> In the app engine application, when looking at the log, there are a >>> number of Info messages but thy all vary, and one warning message (see >>> below): >>> >>> >>> 1. W2015-03-20 10:17:58.727 A problem was encountered with the >>> process that handled this request, causing it to exit. This is likely to >>> cause a new process to be used for the nex >>> >>> >>> Any advice on how to address this error, and if it will cause a >>> potential loss of data (all data has been backed up) in the future? >>> >>> Thanks, >>> >>> ~Lloyd >>> >>> -- >>> -- >>> Post: opend...@googlegroups.com >>> Unsubscribe: opendatakit...@googlegroups.com >>> Options: http://groups.google.com/group/opendatakit?hl=en >>> >>> --- >>> You received this message because you are subscribed to the Google >>> Groups "ODK Community" group. >>> To unsubscribe from this group and stop receiving emails from it, send >>> an email to opendatakit...@googlegroups.com. >>> For more options, visit https://groups.google.com/d/optout. >>> >> >> >> >> -- >> Mitch Sundt >> Software Engineer >> University of Washington >> mitche...@gmail.com >> >

See https://github.com/opendatakit/opendatakit/wiki/Aggregate-AppEngine-Troubleshooting#reparing-a-filled-in-form-submission

Yaw

··· -- Need ODK consultants? Nafundi provides form design, server setup, in-field training, and software development for ODK. Go to https://nafundi.com to get started.

On Mon, May 16, 2016 at 12:55 PM, Brian Musili yalian8@gmail.com wrote:

Mitch,

I have a problem with my aggregate. I am collecting data and trying to post
it using the ODK collect but it is not being transmitted. Secondly when
trying to view the collected data from the aggregate, i can scroll through
the first two pages but on going to the next page i get this

Error: Problem persisting data or accessing data (SELECT * FROM
BUILD_284588833_BASIC_INFORMATION_CURENT_WATER_USE WHERE _TOP_LEVEL_AURI =
uuid:b3358b24-8813-49bc-917e-a7d82801fe14 AND _PARENT_AURI =
uuid:b3358b24-8813-49bc-917e-a7d82801fe14 multiple-choice valueSet is
missing an entry OR has an extra copy of one)

When i query it by GQL

SELECT * FROM BUILD_284588833_BASIC_INFORMATION_CURENT_WATER_USE WHERE
_TOP_LEVEL_AURI = "uuid:b3358b24-8813-49bc-917e-a7d82801fe14" AND
_PARENT_AURI = "uuid:b3358b24-8813-49bc-917e-a7d82801fe14"

i get No entities matched this query

What does this mean and how can i rectify it?

On Saturday, March 21, 2015 at 5:35:18 AM UTC+3, Mitch Sundt wrote:

Several things could be going on:

(1) if many users are all watching the ODK Aggregate webpages at the same
time, they may be competing for processing space and time. Restrict access
to the server and provide nightly reports or publish the data into Google
Fusion Tables and provide access to that.

(2) if you are using "Export to CSV" to produce reports, switch to using
ODK Briefcase to download data to your local computer and then export to CSV
from there. The Export functionality on ODK Aggregate holds all of the
dataset in memory when generating a report. This will not work beyond a
certain size dataset.

(3) increase the "Frontend Instance Class" (machine size) on the AppEngine
( https://appengine.google.com/ ). Choose your application id, then click on
Application Settings on the left sidebar. And select a larger Frontend
Instance Class.

Note that increasing the machine size will increase your costs.

On Fri, Mar 20, 2015 at 10:31 AM, Lloyd Owen Banwart lloyd....@gmail.com wrote:

Hi all,

We have a survey going live at the moment, with data being uploaded on a
daily basis. We are using a server a google app engine deployment of ODK
Aggregate version 1.4.3. It was my intention to update this to the latest
version of Aggregate once this fieldwork was completed.

To date, there has been no issues, but starting today when viewing data
on the "Submission" page of our appspot page for this server, I have been
getting the below error. The error is repetitive, but each time the uuid
case number changes.

"Error: Problem persisting data or accessing data
([ChoiceSubmissionType.getValueFromEntity] SQL: SELECT * FROM
opendatakit.ASUBL_FINAL2_ELG_B_SEC2_MJ_G6_J10C WHERE _PARENT_AURI =
uuid:6219f226-5a47-4501-b439-061dfce52d34 ORDER BY _PARENT_AURI
exception: This
request (2d170ed258bfabee) started at 2015/03/20 16:56:04.949 UTC and was
still
executing at 2015/03/20 16:57:04.779 UTC.)"

In the app engine application, when looking at the log, there are a
number of Info messages but thy all vary, and one warning message (see
below):

W2015-03-20 10:17:58.727 A problem was encountered with the process that
handled this request, causing it to exit. This is likely to cause a new
process to be used for the nex

Any advice on how to address this error, and if it will cause a potential
loss of data (all data has been backed up) in the future?

Thanks,

~Lloyd

--

Post: opend...@googlegroups.com
Unsubscribe: opendatakit...@googlegroups.com
Options: http://groups.google.com/group/opendatakit?hl=en


You received this message because you are subscribed to the Google Groups
"ODK Community" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to opendatakit...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--
Mitch Sundt
Software Engineer
University of Washington
mitche...@gmail.com

--

Post: opendatakit@googlegroups.com
Unsubscribe: opendatakit+unsubscribe@googlegroups.com
Options: http://groups.google.com/group/opendatakit?hl=en


You received this message because you are subscribed to the Google Groups
"ODK Community" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to opendatakit+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

See https://opendatakit.org/use/aggregate/tomcat-install/

search down for "Select and Install Database Server"

or search for "max_allowed_packet" in the text.

··· On Thu, Oct 27, 2016 at 12:41 PM, Colin McCann wrote:

Hi Vincent,

I believe I have encountered this once as well, but am struggling to
remember where the issue lay (apache/tomcat/mysql). My intuition is that it
is mysql. Check your my.cnf (usually located in /etc/mysql) for the
variable max_allowed_packet, and increase it. Mine is currently set to
16MB. And remember to restart mysql after you make the change...

On Thu, Oct 27, 2016 at 4:02 AM, vsamuel@medcol.mw wrote:

Hi Everyone,

I have the following error when I try to upload my forms into the ODK
Aggregate:
Error: Problem persisting data or accessing data
(org.springframework.dao.TransientDataAccessResourceException:
PreparedStatementCallback; SQL [SELECT _URI, _CREATOR_URI_USER,
_CREATION_DATE, _LAST_UPDATE_URI_USER, _LAST_UPDATE_DATE,
_MODEL_VERSION, _UI_VERSION, _IS_COMPLETE, _SUBMISSION_DATE,
_MARKED_AS_COMPLETE_DATE, GRP2_GRP2A_HIVMEDPREG, HOSPITAL, MOMFNM,
GRP2_GRP2A_GRP2B_P1_FREQ, OTH_TRIBE, DEVICEID, GRP1_OTHERNAT,
GRP2_LSTTEST, REPORTDATE, GRP2_RESULT, GRP2_GRP2A_GRP2B_OTHER_P3,
GRP2_GRP2A_GRP2B_OTHER_P2, GRP1_MOMPHONE, GRP1_OTHERWATER,
META_INSTANCE_ID, GRP1_MOMHEIGHT, GRP1_WATER, TRIBE, MOMSNM,
GRP1_MOMWEIGHT2, GRP1_MOMWEIGHT1, GRP1_SMOKERS, GRP1_NATIONALITY,
GRP2_GRP2A_GRP2B_OTHER_P1, GRP2_GRP2A_GRP2B_OTHER_B1, GRP1_CURLIVE,
SIGNED_CONSENT, GRP2_FIRSTTEST, ID_ANCIDNO, GRP1_JOB,
GRP1_CHLDNO, GRP1_JOBTYPE, TODAY, GRP1_INCOME, PAPFNM,
GRP1_OTHERLIVE, GRP1_MOMMARST, GRP2_GRP2A_GRP2B_P3_FREQ,
GRP2_GRP2A_GRP2B_P2_FREQ, CONSENT, PAPSNM, INFO8, INFO7, INFO4,
GRP1_SCHLEV, INFO1, GRP2_GRP2A_GRP2B_WHYSMISS, SURVEY_COMPLETED,
ID_OTHIDNO, GRP1_OTHERTOILET, GRP2_GRP2A_TRADHLR, EVERTEST,
GRP2_GRP2A_GRP2B_MISSHIVMED, GRP1_RMNO, GRP2_GRP2A_GRP2B_B1_FREQ,
ID_IPIDNO, ID_ARTIDNO, QCINITIAL1, SURVEY_DECLINE, VILLAGE,
GRP1_HSEPPLNO, START, ID_PMTCTIDNO, ID_UNQIDNO, END,
GRP2_GRP2A_RESULT2, GRP1_TOILET FROM odk_birthdefects.BIRTH_DEFECTS_CORE
WHERE _MARKED_AS_COMPLETE_DATE > ? AND _IS_COMPLETE = ? ORDER BY
_MARKED_AS_COMPLETE_DATE ASC , _URI ASC ;]; Packet for query is too
large (1508 > 1024). You can change this value on the server by setting the
max_allowed_packet' variable.; nested exception is
com.mysql.jdbc.PacketTooBigException: Packet for query is too large
(1508 > 1024). You can change this value on the server by setting the
max_allowed_packet' variable.)

Any thoughts on why I am having this problem and how I can work it out?

Vincent

--

Post: opendatakit@googlegroups.com
Unsubscribe: opendatakit+unsubscribe@googlegroups.com
Options: http://groups.google.com/group/opendatakit?hl=en


You received this message because you are subscribed to a topic in the
Google Groups "ODK Community" group.
To unsubscribe from this topic, visit https://groups.google.com/d/to
pic/opendatakit/gF24RHrltGs/unsubscribe.
To unsubscribe from this group and all its topics, send an email to
opendatakit+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--
Colin

--

Post: opendatakit@googlegroups.com
Unsubscribe: opendatakit+unsubscribe@googlegroups.com
Options: http://groups.google.com/group/opendatakit?hl=en


You received this message because you are subscribed to the Google Groups
"ODK Community" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to opendatakit+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--
Mitch Sundt
Software Engineer
University of Washington
mitchellsundt@gmail.com

And ODK Briefcase works to download the data, or does it give a failure?

If Briefcase is working OK without failures, and if you have a paid
instance, I would contact Google support, as this is a repeatable issue.
You should probably CC me on the support request you open with them, and
grant me developer access to the instance (and let me know the instance
URL) https://code.google.com/p/opendatakit/wiki/AppEngineTroubleshooting

··· On Mon, Mar 23, 2015 at 7:20 AM, Lloyd Owen Banwart <lloyd.banwart@gmail.com wrote:

To follow upon this post.

I had decreased access to the server for 24 hours to two users, and this
did not resolve the issue of receiving the below error. I then steadily
increased the frontend instance Class from 256 to 512 to 1024. In each
case, the error still was displayed on the ODK server. This error only
displays when viewing the data on the "Submissions tab" and under "Filter
Submissions" (no filter is applied). The error is repetitive, meaning that
when I leave the browser open, the error will display, instances may (or
may not) be shown in the submissions tab, then the error will pop up again.
Each time the error displays the error appears identical other than the
uuid, specific Q it is referencing, and start and end time of request.
I've tracked most uuids that have shown in the errors, and cross referenced
this with data obtained after the error has been displayed. All instances
displayed during errors are in the data which have been downloaded, and the
instances appear unaffected.

When accessing the data (via the "Export" function or via ODK Briefcase)
there appears to be no data loss when check against field-records of
uploaded instances compared to instances obtained from the server. Other
than being a annoying, and limiting any quick review of recently submitted
data via the server, I don't see any major issues at the moment with the
error -- but would love to get to the bottom of it.

Second example of error:
Error: Problem persisting data or accessing data
([ChoiceSubmissionType.getValueFromEntity] SQL: SELECT * FROM
opendatakit.ASUBL_FINAL2_ELG_B_SEC2_SEC3_MM_G3_LABELM3_2 WHERE _PARENT_AURI
= uuid:e1ba7512-9c8e-47d1-8458-0ddc1935642b ORDER BY _PARENT_AURI
exception: This request (f44144cff1ba1b4f) started at 2015/03/23
13:51:54.645 UTC and was still executing at 2015/03/23 13:52:54.505 UTC.)

On Monday, March 23, 2015 at 9:39:14 AM UTC-4, Lloyd Owen Banwart wrote:

Mitch. Thanks (as always) for the clear and timely feedback.

  1. I've limited access, but it was a total of 3 with access before (now
    it is a total of 2) to the server.

  2. We use ODK briefcase (as we avail of the Stata ODKmeta program to
    provide value/var labels and restructure data sets from repeat groups).

  3. Re: Increasing the machine size; this is one of the smaller data
    collections we have done. The survey is large, but we are collecting data
    from less than 2,000 households --- is there a reason this issue would
    arise now and it hasn't in the past? Regardless, I'm waiting to see if
    limiting access to the server addresses this issue, if it does not I will
    increase the machine size.

I'll post progress here in the event someone else faces this issue in the
future. Thanks again.

~Lloyd

On Friday, March 20, 2015 at 10:35:18 PM UTC-4, Mitch Sundt wrote:

Several things could be going on:

(1) if many users are all watching the ODK Aggregate webpages at the
same time, they may be competing for processing space and time. Restrict
access to the server and provide nightly reports or publish the data into
Google Fusion Tables and provide access to that.

(2) if you are using "Export to CSV" to produce reports, switch to using
ODK Briefcase to download data to your local computer and then export to
CSV from there. The Export functionality on ODK Aggregate holds all of the
dataset in memory when generating a report. This will not work beyond a
certain size dataset.

(3) increase the "Frontend Instance Class" (machine size) on the
AppEngine ( https://appengine.google.com/ ). Choose your application
id, then click on Application Settings on the left sidebar. And select a
larger Frontend Instance Class.

Note that increasing the machine size will increase your costs.

On Fri, Mar 20, 2015 at 10:31 AM, Lloyd Owen Banwart < lloyd....@gmail.com> wrote:

Hi all,

We have a survey going live at the moment, with data being uploaded on
a daily basis. We are using a server a google app engine deployment of ODK
Aggregate version 1.4.3. It was my intention to update this to the latest
version of Aggregate once this fieldwork was completed.

To date, there has been no issues, but starting today when viewing data
on the "Submission" page of our appspot page for this server, I have been
getting the below error. The error is repetitive, but each time the uuid
case number changes.

"Error: Problem persisting data or accessing data
([ChoiceSubmissionType.getValueFromEntity] SQL: SELECT * FROM
opendatakit.ASUBL_FINAL2_ELG_B_SEC2_MJ_G6_J10C WHERE _PARENT_AURI =
uuid:6219f226-5a47-4501-b439-061dfce52d34 ORDER BY _PARENT_AURI
exception: This
request (2d170ed258bfabee) started at 2015/03/20 16:56:04.949 UTC and
was still
executing at 2015/03/20 16:57:04.779 UTC.)"

In the app engine application, when looking at the log, there are a
number of Info messages but thy all vary, and one warning message (see
below):

  1. W2015-03-20 10:17:58.727 A problem was encountered with the
    process that handled this request, causing it to exit. This is likely to
    cause a new process to be used for the nex

Any advice on how to address this error, and if it will cause a
potential loss of data (all data has been backed up) in the future?

Thanks,

~Lloyd

--

Post: opend...@googlegroups.com
Unsubscribe: opendatakit...@googlegroups.com
Options: http://groups.google.com/group/opendatakit?hl=en


You received this message because you are subscribed to the Google
Groups "ODK Community" group.
To unsubscribe from this group and stop receiving emails from it, send
an email to opendatakit...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--
Mitch Sundt
Software Engineer
University of Washington
mitche...@gmail.com

--
--
Post: opendatakit@googlegroups.com
Unsubscribe: opendatakit+unsubscribe@googlegroups.com
Options: http://groups.google.com/group/opendatakit?hl=en


You received this message because you are subscribed to the Google Groups
"ODK Community" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to opendatakit+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--
Mitch Sundt
Software Engineer
University of Washington
mitchellsundt@gmail.com

Hi all,

I have encountered the above issue and have been trying to get it resolved.

Error: Problem persisting data or accessing data ([ChoiceSubmissionType.getValueFromEntity] SQL: SELECT * FROM opendatakit.F6_V14_ECD4A WHERE _PARENT_AURI = uuid:f6f98bdc-f55a-437a-9c3e-54718828ce3a ORDER BY _PARENT_AURI exception: This request (380af43f8586449b) started at 2016/05/24 20:42:38.199 UTC and was still executing at 2016/05/24 20:43:38.520 UTC.)

I can confirm that if I lower the "Submissions per page" to eg 10, I no longer encounter the error.

However, I have tried the steps outlined in: https://github.com/opendatakit/opendatakit/wiki/Aggregate-AppEngine-Troubleshooting#reparing-a-filled-in-form-submission
without success.

From the error logs:
org.apache.commons.logging.impl.Jdk14Logger log: Retrying fetch with a smaller chunk size: [ChoiceSubmissionType.getValueFromEntity] SQL: SELECT * FROM opendatakit.F6_V14_ECD4A WHERE _PARENT_AURI = uuid:f6f98bdc-f55a-437a-9c3e-54718828ce3a ORDER BY _PARENT_AURI exception: Task was cancelled. (Jdk14Logger.java:87)

When I run the Query by GQL:
SELECT * FROM opendatakit.F6_V14_ECD4A WHERE _PARENT_AURI = "uuid:f6f98bdc-f55a-437a-9c3e-54718828ce3a"
I get no results returned.
To confirm, I ran
SELECT * FROM opendatakit.F6_V14_ECD4A
And manually scanned for the uuid, couldn't find it

I'm not clear on why it would be querying for a UUID that doesn't exist, but it's concerning. Any thoughts on how I should proceed?

Thanks in advance

ODK Briefcase is working without failures. I do have paid instances.

··· On Tuesday, March 24, 2015 at 10:45:49 AM UTC-4, Mitch Sundt wrote: > > And ODK Briefcase works to download the data, or does it give a failure? > > If Briefcase is working OK without failures, and if you have a paid > instance, I would contact Google support, as this is a repeatable issue. > You should probably CC me on the support request you open with them, and > grant me developer access to the instance (and let me know the instance > URL) https://code.google.com/p/opendatakit/wiki/AppEngineTroubleshooting > > > On Mon, Mar 23, 2015 at 7:20 AM, Lloyd Owen Banwart <lloyd....@gmail.com > wrote: > >> To follow upon this post. >> >> I had decreased access to the server for 24 hours to two users, and this >> did not resolve the issue of receiving the below error. I then steadily >> increased the frontend instance Class from 256 to 512 to 1024. In each >> case, the error still was displayed on the ODK server. This error only >> displays when viewing the data on the "Submissions tab" and under "Filter >> Submissions" (no filter is applied). The error is repetitive, meaning that >> when I leave the browser open, the error will display, instances may (or >> may not) be shown in the submissions tab, then the error will pop up again. >> Each time the error displays the error appears identical other than the >> uuid, specific Q it is referencing, and start and end time of request. >> I've tracked most uuids that have shown in the errors, and cross referenced >> this with data obtained after the error has been displayed. All instances >> displayed during errors are in the data which have been downloaded, and the >> instances appear unaffected. >> >> When accessing the data (via the "Export" function or via ODK Briefcase) >> there appears to be no data loss when check against field-records of >> uploaded instances compared to instances obtained from the server. Other >> than being a annoying, and limiting any quick review of recently submitted >> data via the server, I don't see any major issues at the moment with the >> error -- but would love to get to the bottom of it. >> >> Second example of error: >> Error: Problem persisting data or accessing data >> ([ChoiceSubmissionType.getValueFromEntity] SQL: SELECT * FROM >> opendatakit.ASUBL_FINAL2_ELG_B_SEC2_SEC3_MM_G3_LABELM3_2 WHERE _PARENT_AURI >> = uuid:e1ba7512-9c8e-47d1-8458-0ddc1935642b ORDER BY _PARENT_AURI >> exception: This request (f44144cff1ba1b4f) started at 2015/03/23 >> 13:51:54.645 UTC and was still executing at 2015/03/23 13:52:54.505 UTC.) >> >> >> >> On Monday, March 23, 2015 at 9:39:14 AM UTC-4, Lloyd Owen Banwart wrote: >>> >>> Mitch. Thanks (as always) for the clear and timely feedback. >>> >>> 1) I've limited access, but it was a total of 3 with access before (now >>> it is a total of 2) to the server. >>> >>> 2) We use ODK briefcase (as we avail of the Stata ODKmeta program to >>> provide value/var labels and restructure data sets from repeat groups). >>> >>> 3) Re: Increasing the machine size; this is one of the smaller data >>> collections we have done. The survey is large, but we are collecting data >>> from less than 2,000 households --- is there a reason this issue would >>> arise now and it hasn't in the past? Regardless, I'm waiting to see if >>> limiting access to the server addresses this issue, if it does not I will >>> increase the machine size. >>> >>> I'll post progress here in the event someone else faces this issue in >>> the future. Thanks again. >>> >>> ~Lloyd >>> >>> On Friday, March 20, 2015 at 10:35:18 PM UTC-4, Mitch Sundt wrote: >>>> >>>> Several things could be going on: >>>> >>>> (1) if many users are all watching the ODK Aggregate webpages at the >>>> same time, they may be competing for processing space and time. Restrict >>>> access to the server and provide nightly reports or publish the data into >>>> Google Fusion Tables and provide access to that. >>>> >>>> (2) if you are using "Export to CSV" to produce reports, switch to >>>> using ODK Briefcase to download data to your local computer and then export >>>> to CSV from there. The Export functionality on ODK Aggregate holds all of >>>> the dataset in memory when generating a report. This will not work beyond a >>>> certain size dataset. >>>> >>>> (3) increase the "Frontend Instance Class" (machine size) on the >>>> AppEngine ( https://appengine.google.com/ ). Choose your application >>>> id, then click on Application Settings on the left sidebar. And select a >>>> larger Frontend Instance Class. >>>> >>>> Note that increasing the machine size will increase your costs. >>>> >>>> >>>> On Fri, Mar 20, 2015 at 10:31 AM, Lloyd Owen Banwart < lloyd....@gmail.com> wrote: >>>> >>>>> Hi all, >>>>> >>>>> We have a survey going live at the moment, with data being uploaded on >>>>> a daily basis. We are using a server a google app engine deployment of ODK >>>>> Aggregate version 1.4.3. It was my intention to update this to the latest >>>>> version of Aggregate once this fieldwork was completed. >>>>> >>>>> To date, there has been no issues, but starting today when viewing >>>>> data on the "Submission" page of our appspot page for this server, I have >>>>> been getting the below error. The error is repetitive, but each time the >>>>> uuid case number changes. >>>>> >>>>> "Error: Problem persisting data or accessing data >>>>> ([ChoiceSubmissionType.getValueFromEntity] SQL: SELECT * FROM >>>>> opendatakit.ASUBL_FINAL2_ELG_B_SEC2_MJ_G6_J10C WHERE _PARENT_AURI = >>>>> uuid:6219f226-5a47-4501-b439-061dfce52d34 ORDER BY _PARENT_AURI >>>>> exception: This >>>>> request (2d170ed258bfabee) started at 2015/03/20 16:56:04.949 UTC and >>>>> was still >>>>> executing at 2015/03/20 16:57:04.779 UTC.)" >>>>> >>>>> >>>>> In the app engine application, when looking at the log, there are a >>>>> number of Info messages but thy all vary, and one warning message (see >>>>> below): >>>>> >>>>> >>>>> 1. W2015-03-20 10:17:58.727 A problem was encountered with the >>>>> process that handled this request, causing it to exit. This is likely to >>>>> cause a new process to be used for the nex >>>>> >>>>> >>>>> Any advice on how to address this error, and if it will cause a >>>>> potential loss of data (all data has been backed up) in the future? >>>>> >>>>> Thanks, >>>>> >>>>> ~Lloyd >>>>> >>>>> -- >>>>> -- >>>>> Post: opend...@googlegroups.com >>>>> Unsubscribe: opendatakit...@googlegroups.com >>>>> Options: http://groups.google.com/group/opendatakit?hl=en >>>>> >>>>> --- >>>>> You received this message because you are subscribed to the Google >>>>> Groups "ODK Community" group. >>>>> To unsubscribe from this group and stop receiving emails from it, send >>>>> an email to opendatakit...@googlegroups.com. >>>>> For more options, visit https://groups.google.com/d/optout. >>>>> >>>> >>>> >>>> >>>> -- >>>> Mitch Sundt >>>> Software Engineer >>>> University of Washington >>>> mitche...@gmail.com >>>> >>> -- >> -- >> Post: opend...@googlegroups.com >> Unsubscribe: opendatakit...@googlegroups.com >> Options: http://groups.google.com/group/opendatakit?hl=en >> >> --- >> You received this message because you are subscribed to the Google Groups >> "ODK Community" group. >> To unsubscribe from this group and stop receiving emails from it, send an >> email to opendatakit...@googlegroups.com . >> For more options, visit https://groups.google.com/d/optout. >> > > > > -- > Mitch Sundt > Software Engineer > University of Washington > mitche...@gmail.com >

"Task was cancelled" indicates a problem in the Google infrastructure. It
does not necessarily mean that your datastore has a problem with this
record (Google is stuck and not returning success or failure).

You can open an issue with Google to find and fix this problem (this may
only be available to accounts with billing enabled), or you can wait. At
some point, they will likely fix it. I am unfortunately pessimistic about
their responsiveness to problem tickets (in your ticket, give them the GQL
you are trying to run).

If you were inserting a data record when Google began having trouble, it is
likely that once Google fixes the issue, you will then need to follow the
steps to fix the resulting data corruption, which may or may not be related
to this record.

··· On Tue, May 24, 2016 at 1:49 PM, wrote:

Hi all,

I have encountered the above issue and have been trying to get it resolved.

Error: Problem persisting data or accessing data
([ChoiceSubmissionType.getValueFromEntity] SQL: SELECT * FROM
opendatakit.F6_V14_ECD4A WHERE _PARENT_AURI =
uuid:f6f98bdc-f55a-437a-9c3e-54718828ce3a ORDER BY _PARENT_AURI exception:
This request (380af43f8586449b) started at 2016/05/24 20:42:38.199 UTC and
was still executing at 2016/05/24 20:43:38.520 UTC.)

I can confirm that if I lower the "Submissions per page" to eg 10, I no
longer encounter the error.

However, I have tried the steps outlined in:
https://github.com/opendatakit/opendatakit/wiki/Aggregate-AppEngine-Troubleshooting#reparing-a-filled-in-form-submission
without success.

From the error logs:
org.apache.commons.logging.impl.Jdk14Logger log: Retrying fetch with a
smaller chunk size: [ChoiceSubmissionType.getValueFromEntity] SQL: SELECT *
FROM opendatakit.F6_V14_ECD4A WHERE _PARENT_AURI =
uuid:f6f98bdc-f55a-437a-9c3e-54718828ce3a ORDER BY _PARENT_AURI exception:
Task was cancelled. (Jdk14Logger.java:87)

When I run the Query by GQL:
SELECT * FROM opendatakit.F6_V14_ECD4A WHERE _PARENT_AURI =
"uuid:f6f98bdc-f55a-437a-9c3e-54718828ce3a"
I get no results returned.
To confirm, I ran
SELECT * FROM opendatakit.F6_V14_ECD4A
And manually scanned for the uuid, couldn't find it

I'm not clear on why it would be querying for a UUID that doesn't exist,
but it's concerning. Any thoughts on how I should proceed?

Thanks in advance

--

Post: opendatakit@googlegroups.com
Unsubscribe: opendatakit+unsubscribe@googlegroups.com
Options: http://groups.google.com/group/opendatakit?hl=en


You received this message because you are subscribed to the Google Groups
"ODK Community" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to opendatakit+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--
Mitch Sundt
Software Engineer
University of Washington
mitchellsundt@gmail.com

Re-posting an offline reply from Mitch here (I took the conversation
offline as I granted him developer access) that brings this issue to a
close.

"OK. Looking at the logs, I think everything is working as designed.

*The form being displayed on the Submissions tab takes more than 60 seconds
to completely fetch 100 rows from the database. *

AppEngine imposes a 60-second limit, after which it aggressively
terminates any outstanding database calls, and the webserver detects and
returns that error up to the webpage. Issuing those queries directly in
the management console (
https://console.developers.google.com/project/tangointernational-3/datastore/query
) works fine and the results look like they should.

If you change the "Submissions per Page" to something less (e.g., 50), the
error should go away and you should be able to see the data in the
submissions tab.

If memory serves, that setting choice is not remembered once you leave the
page, so you would always have to re-enter it."

I can confirm that reducing the submissions per page seems to negate this
error.

Thank you Mitch for to diligent follow-up and assistance.

··· On Tuesday, March 24, 2015 at 10:58:28 AM UTC-4, Lloyd Owen Banwart wrote: > > ODK Briefcase is working without failures. I do have paid instances. > > On Tuesday, March 24, 2015 at 10:45:49 AM UTC-4, Mitch Sundt wrote: >> >> And ODK Briefcase works to download the data, or does it give a failure? >> >> If Briefcase is working OK without failures, and if you have a paid >> instance, I would contact Google support, as this is a repeatable issue. >> You should probably CC me on the support request you open with them, and >> grant me developer access to the instance (and let me know the instance >> URL) https://code.google.com/p/opendatakit/wiki/AppEngineTroubleshooting >> >> >> On Mon, Mar 23, 2015 at 7:20 AM, Lloyd Owen Banwart wrote: >> >>> To follow upon this post. >>> >>> I had decreased access to the server for 24 hours to two users, and this >>> did not resolve the issue of receiving the below error. I then steadily >>> increased the frontend instance Class from 256 to 512 to 1024. In each >>> case, the error still was displayed on the ODK server. This error only >>> displays when viewing the data on the "Submissions tab" and under "Filter >>> Submissions" (no filter is applied). The error is repetitive, meaning that >>> when I leave the browser open, the error will display, instances may (or >>> may not) be shown in the submissions tab, then the error will pop up again. >>> Each time the error displays the error appears identical other than the >>> uuid, specific Q it is referencing, and start and end time of request. >>> I've tracked most uuids that have shown in the errors, and cross referenced >>> this with data obtained after the error has been displayed. All instances >>> displayed during errors are in the data which have been downloaded, and the >>> instances appear unaffected. >>> >>> When accessing the data (via the "Export" function or via ODK Briefcase) >>> there appears to be no data loss when check against field-records of >>> uploaded instances compared to instances obtained from the server. Other >>> than being a annoying, and limiting any quick review of recently submitted >>> data via the server, I don't see any major issues at the moment with the >>> error -- but would love to get to the bottom of it. >>> >>> Second example of error: >>> Error: Problem persisting data or accessing data >>> ([ChoiceSubmissionType.getValueFromEntity] SQL: SELECT * FROM >>> opendatakit.ASUBL_FINAL2_ELG_B_SEC2_SEC3_MM_G3_LABELM3_2 WHERE _PARENT_AURI >>> = uuid:e1ba7512-9c8e-47d1-8458-0ddc1935642b ORDER BY _PARENT_AURI >>> exception: This request (f44144cff1ba1b4f) started at 2015/03/23 >>> 13:51:54.645 UTC and was still executing at 2015/03/23 13:52:54.505 UTC.) >>> >>> >>> >>> On Monday, March 23, 2015 at 9:39:14 AM UTC-4, Lloyd Owen Banwart wrote: >>>> >>>> Mitch. Thanks (as always) for the clear and timely feedback. >>>> >>>> 1) I've limited access, but it was a total of 3 with access before (now >>>> it is a total of 2) to the server. >>>> >>>> 2) We use ODK briefcase (as we avail of the Stata ODKmeta program to >>>> provide value/var labels and restructure data sets from repeat groups). >>>> >>>> 3) Re: Increasing the machine size; this is one of the smaller data >>>> collections we have done. The survey is large, but we are collecting data >>>> from less than 2,000 households --- is there a reason this issue would >>>> arise now and it hasn't in the past? Regardless, I'm waiting to see if >>>> limiting access to the server addresses this issue, if it does not I will >>>> increase the machine size. >>>> >>>> I'll post progress here in the event someone else faces this issue in >>>> the future. Thanks again. >>>> >>>> ~Lloyd >>>> >>>> On Friday, March 20, 2015 at 10:35:18 PM UTC-4, Mitch Sundt wrote: >>>>> >>>>> Several things could be going on: >>>>> >>>>> (1) if many users are all watching the ODK Aggregate webpages at the >>>>> same time, they may be competing for processing space and time. Restrict >>>>> access to the server and provide nightly reports or publish the data into >>>>> Google Fusion Tables and provide access to that. >>>>> >>>>> (2) if you are using "Export to CSV" to produce reports, switch to >>>>> using ODK Briefcase to download data to your local computer and then export >>>>> to CSV from there. The Export functionality on ODK Aggregate holds all of >>>>> the dataset in memory when generating a report. This will not work beyond a >>>>> certain size dataset. >>>>> >>>>> (3) increase the "Frontend Instance Class" (machine size) on the >>>>> AppEngine ( https://appengine.google.com/ ). Choose your application >>>>> id, then click on Application Settings on the left sidebar. And select a >>>>> larger Frontend Instance Class. >>>>> >>>>> Note that increasing the machine size will increase your costs. >>>>> >>>>> >>>>> On Fri, Mar 20, 2015 at 10:31 AM, Lloyd Owen Banwart < lloyd....@gmail.com> wrote: >>>>> >>>>>> Hi all, >>>>>> >>>>>> We have a survey going live at the moment, with data being uploaded >>>>>> on a daily basis. We are using a server a google app engine deployment of >>>>>> ODK Aggregate version 1.4.3. It was my intention to update this to the >>>>>> latest version of Aggregate once this fieldwork was completed. >>>>>> >>>>>> To date, there has been no issues, but starting today when viewing >>>>>> data on the "Submission" page of our appspot page for this server, I have >>>>>> been getting the below error. The error is repetitive, but each time the >>>>>> uuid case number changes. >>>>>> >>>>>> "Error: Problem persisting data or accessing data >>>>>> ([ChoiceSubmissionType.getValueFromEntity] SQL: SELECT * FROM >>>>>> opendatakit.ASUBL_FINAL2_ELG_B_SEC2_MJ_G6_J10C WHERE _PARENT_AURI = >>>>>> uuid:6219f226-5a47-4501-b439-061dfce52d34 ORDER BY _PARENT_AURI >>>>>> exception: This >>>>>> request (2d170ed258bfabee) started at 2015/03/20 16:56:04.949 UTC and >>>>>> was still >>>>>> executing at 2015/03/20 16:57:04.779 UTC.)" >>>>>> >>>>>> >>>>>> In the app engine application, when looking at the log, there are a >>>>>> number of Info messages but thy all vary, and one warning message (see >>>>>> below): >>>>>> >>>>>> >>>>>> 1. W2015-03-20 10:17:58.727 A problem was encountered with the >>>>>> process that handled this request, causing it to exit. This is likely to >>>>>> cause a new process to be used for the nex >>>>>> >>>>>> >>>>>> Any advice on how to address this error, and if it will cause a >>>>>> potential loss of data (all data has been backed up) in the future? >>>>>> >>>>>> Thanks, >>>>>> >>>>>> ~Lloyd >>>>>> >>>>>> -- >>>>>> -- >>>>>> Post: opend...@googlegroups.com >>>>>> Unsubscribe: opendatakit...@googlegroups.com >>>>>> Options: http://groups.google.com/group/opendatakit?hl=en >>>>>> >>>>>> --- >>>>>> You received this message because you are subscribed to the Google >>>>>> Groups "ODK Community" group. >>>>>> To unsubscribe from this group and stop receiving emails from it, >>>>>> send an email to opendatakit...@googlegroups.com. >>>>>> For more options, visit https://groups.google.com/d/optout. >>>>>> >>>>> >>>>> >>>>> >>>>> -- >>>>> Mitch Sundt >>>>> Software Engineer >>>>> University of Washington >>>>> mitche...@gmail.com >>>>> >>>> -- >>> -- >>> Post: opend...@googlegroups.com >>> Unsubscribe: opendatakit...@googlegroups.com >>> Options: http://groups.google.com/group/opendatakit?hl=en >>> >>> --- >>> You received this message because you are subscribed to the Google >>> Groups "ODK Community" group. >>> To unsubscribe from this group and stop receiving emails from it, send >>> an email to opendatakit...@googlegroups.com. >>> For more options, visit https://groups.google.com/d/optout. >>> >> >> >> >> -- >> Mitch Sundt >> Software Engineer >> University of Washington >> mitche...@gmail.com >> >

Thanks Mitch,

So you're saying that the error I see in Aggregate:

Error: Problem persisting data or accessing data ([ChoiceSubmissionType.getValueFromEntity] SQL: SELECT * FROM opendatakit.F6_V14_ECD4A WHERE _PARENT_AURI = uuid:f6f98bdc-f55a-437a-9c3e-54718828ce3a ORDER BY _PARENT_AURI exception: This request (380af43f8586449b) started at 2016/05/24 20:42:38.199 UTC and was still executing at 2016/05/24 20:43:38.520 UTC.)

is due to an issue with Google infrastructure and therefore not something that I can directly address (other than issuing a ticket)?

··· On Tuesday, May 24, 2016 at 5:30:45 PM UTC-4, Mitch Sundt wrote: > "Task was cancelled" indicates a problem in the Google infrastructure. It does not necessarily mean that your datastore has a problem with this record (Google is stuck and not returning success or failure). > > > You can open an issue with Google to find and fix this problem (this may only be available to accounts with billing enabled), or you can wait. At some point, they will likely fix it. I am unfortunately pessimistic about their responsiveness to problem tickets (in your ticket, give them the GQL you are trying to run). > > > > If you were inserting a data record when Google began having trouble, it is likely that once Google fixes the issue, you will then need to follow the steps to fix the resulting data corruption, which may or may not be related to this record. > > > > > On Tue, May 24, 2016 at 1:49 PM, wrote: > Hi all, > > > > I have encountered the above issue and have been trying to get it resolved. > > > > Error: Problem persisting data or accessing data ([ChoiceSubmissionType.getValueFromEntity] SQL: SELECT * FROM opendatakit.F6_V14_ECD4A WHERE _PARENT_AURI = uuid:f6f98bdc-f55a-437a-9c3e-54718828ce3a ORDER BY _PARENT_AURI exception: This request (380af43f8586449b) started at 2016/05/24 20:42:38.199 UTC and was still executing at 2016/05/24 20:43:38.520 UTC.) > > > > > > > > I can confirm that if I lower the "Submissions per page" to eg 10, I no longer encounter the error. > > > > However, I have tried the steps outlined in: https://github.com/opendatakit/opendatakit/wiki/Aggregate-AppEngine-Troubleshooting#reparing-a-filled-in-form-submission > > without success. > > > > From the error logs: > > org.apache.commons.logging.impl.Jdk14Logger log: Retrying fetch with a smaller chunk size: [ChoiceSubmissionType.getValueFromEntity] SQL: SELECT * FROM opendatakit.F6_V14_ECD4A WHERE _PARENT_AURI = uuid:f6f98bdc-f55a-437a-9c3e-54718828ce3a ORDER BY _PARENT_AURI exception: Task was cancelled. (Jdk14Logger.java:87) > > > > When I run the Query by GQL: > > SELECT * FROM `opendatakit.F6_V14_ECD4A` WHERE _PARENT_AURI = "uuid:f6f98bdc-f55a-437a-9c3e-54718828ce3a" > > I get no results returned. > > To confirm, I ran > > SELECT * FROM `opendatakit.F6_V14_ECD4A` > > And manually scanned for the uuid, couldn't find it > > > > > > I'm not clear on why it would be querying for a UUID that doesn't exist, but it's concerning. Any thoughts on how I should proceed? > > > > Thanks in advance > > > > > > -- > > -- > > Post: opend...@googlegroups.com > > Unsubscribe: opendatakit...@googlegroups.com > > Options: http://groups.google.com/group/opendatakit?hl=en > > > > --- > > You received this message because you are subscribed to the Google Groups "ODK Community" group. > > To unsubscribe from this group and stop receiving emails from it, send an email to opendatakit...@googlegroups.com. > > For more options, visit https://groups.google.com/d/optout. > > > > > > -- > > Mitch Sundt > Software Engineer > University of Washington > mitche...@gmail.com

Thanks

Thank you Lloyd and Mitch for getting to the bottom of this. We're having the same issue and are relieved to hear that this error message doesn't indicate any loss of data!

··· On Wednesday, March 25, 2015 at 12:42:07 AM UTC+9, Lloyd Owen Banwart wrote: > Re-posting an offline reply from Mitch here (I took the conversation offline as I granted him developer access) that brings this issue to a close. > > > > > "OK. Looking at the logs, I think everything is working as designed. > > > The form being displayed on the Submissions tab takes more than 60 seconds to completely fetch 100 rows from the database. > > > AppEngine imposes a 60-second limit, after which it aggressively terminates any outstanding database calls, and the webserver detects and returns that error up to the webpage. Issuing those queries directly in the management console ( https://console.developers.google.com/project/tangointernational-3/datastore/query ) works fine and the results look like they should. > > > If you change the "Submissions per Page" to something less (e.g., 50), the error should go away and you should be able to see the data in the submissions tab. > > > If memory serves, that setting choice is not remembered once you leave the page, so you would always have to re-enter it." > > > I can confirm that reducing the submissions per page seems to negate this error. > > > Thank you Mitch for to diligent follow-up and assistance. > > On Tuesday, March 24, 2015 at 10:58:28 AM UTC-4, Lloyd Owen Banwart wrote: > ODK Briefcase is working without failures. I do have paid instances. > > On Tuesday, March 24, 2015 at 10:45:49 AM UTC-4, Mitch Sundt wrote: > > And ODK Briefcase works to download the data, or does it give a failure? > > If Briefcase is working OK without failures, and if you have a paid instance, I would contact Google support, as this is a repeatable issue. You should probably CC me on the support request you open with them, and grant me developer access to the instance (and let me know the instance URL) https://code.google.com/p/opendatakit/wiki/AppEngineTroubleshooting > > > > > On Mon, Mar 23, 2015 at 7:20 AM, Lloyd Owen Banwart wrote: > > To follow upon this post. > > > I had decreased access to the server for 24 hours to two users, and this did not resolve the issue of receiving the below error. I then steadily increased the frontend instance Class from 256 to 512 to 1024. In each case, the error still was displayed on the ODK server. This error only displays when viewing the data on the "Submissions tab" and under "Filter Submissions" (no filter is applied). The error is repetitive, meaning that when I leave the browser open, the error will display, instances may (or may not) be shown in the submissions tab, then the error will pop up again. Each time the error displays the error appears identical other than the uuid, specific Q it is referencing, and start and end time of request. I've tracked most uuids that have shown in the errors, and cross referenced this with data obtained after the error has been displayed. All instances displayed during errors are in the data which have been downloaded, and the instances appear unaffected. > > > When accessing the data (via the "Export" function or via ODK Briefcase) there appears to be no data loss when check against field-records of uploaded instances compared to instances obtained from the server. Other than being a annoying, and limiting any quick review of recently submitted data via the server, I don't see any major issues at the moment with the error -- but would love to get to the bottom of it. > > > Second example of error: > Error: Problem persisting > data or accessing data ([ChoiceSubmissionType.getValueFromEntity] SQL: SELECT * > FROM opendatakit.ASUBL_FINAL2_ELG_B_SEC2_SEC3_MM_G3_LABELM3_2 WHERE > _PARENT_AURI = uuid:e1ba7512-9c8e-47d1-8458-0ddc1935642b ORDER BY _PARENT_AURI > exception: This request (f44144cff1ba1b4f) started at 2015/03/23 13:51:54.645 > UTC and was still executing at 2015/03/23 13:52:54.505 UTC.) > > > > > > > On Monday, March 23, 2015 at 9:39:14 AM UTC-4, Lloyd Owen Banwart wrote: > Mitch. Thanks (as always) for the clear and timely feedback. > > > 1) I've limited access, but it was a total of 3 with access before (now it is a total of 2) to the server. > > > 2) We use ODK briefcase (as we avail of the Stata ODKmeta program to provide value/var labels and restructure data sets from repeat groups). > > > 3) Re: Increasing the machine size; this is one of the smaller data collections we have done. The survey is large, but we are collecting data from less than 2,000 households --- is there a reason this issue would arise now and it hasn't in the past? Regardless, I'm waiting to see if limiting access to the server addresses this issue, if it does not I will increase the machine size. > > > I'll post progress here in the event someone else faces this issue in the future. Thanks again. > > > ~Lloyd > > On Friday, March 20, 2015 at 10:35:18 PM UTC-4, Mitch Sundt wrote: > > > > Several things could be going on: > > (1) if many users are all watching the ODK Aggregate webpages at the same time, they may be competing for processing space and time. Restrict access to the server and provide nightly reports or publish the data into Google Fusion Tables and provide access to that. > > > (2) if you are using "Export to CSV" to produce reports, switch to using ODK Briefcase to download data to your local computer and then export to CSV from there. The Export functionality on ODK Aggregate holds all of the dataset in memory when generating a report. This will not work beyond a certain size dataset. > > (3) increase the "Frontend Instance Class" (machine size) on the AppEngine ( https://appengine.google.com/ ). Choose your application id, then click on Application Settings on the left sidebar. And select a larger Frontend Instance Class. > > Note that increasing the machine size will increase your costs. > > > > > On Fri, Mar 20, 2015 at 10:31 AM, Lloyd Owen Banwart wrote: > > Hi all, > > > We have a survey going live at the moment, with data being uploaded on a daily basis. We are using a server a google app engine deployment of ODK Aggregate version 1.4.3. It was my intention to update this to the latest version of Aggregate once this fieldwork was completed. > > > To date, there has been no issues, but starting today when viewing data on the "Submission" page of our appspot page for this server, I have been getting the below error. The error is repetitive, but each time the uuid case number changes. > > > "Error: Problem persisting data or accessing data([ChoiceSubmissionType.getValueFromEntity] SQL: SELECT * FROM > opendatakit.ASUBL_FINAL2_ELG_B_SEC2_MJ_G6_J10C WHERE _PARENT_AURI = > uuid:6219f226-5a47-4501-b439-061dfce52d34 ORDER BY _PARENT_AURI exception: This > request (2d170ed258bfabee) started at 2015/03/20 16:56:04.949 UTC and was still > executing at 2015/03/20 16:57:04.779 UTC.)" > > > In the app engine application, when looking at the log, there are a number of Info messages but thy all vary, and one warning message (see below): > > > > W2015-03-20 10:17:58.727 A problem was encountered with the process that handled this request, causing it to exit. This is likely to cause a new process to be used for the nex > > > Any advice on how to address this error, and if it will cause a potential loss of data (all data has been backed up) in the future? > > > Thanks, > > > ~Lloyd > > > > > > > -- > > -- > > Post: opend...@googlegroups.com > > Unsubscribe: opendatakit...@googlegroups.com > > Options: http://groups.google.com/group/opendatakit?hl=en > > > > --- > > You received this message because you are subscribed to the Google Groups "ODK Community" group. > > To unsubscribe from this group and stop receiving emails from it, send an email to opendatakit...@googlegroups.com. > > For more options, visit https://groups.google.com/d/optout. > > > > > -- > > Mitch Sundt > Software Engineer > University of Washington > mitche...@gmail.com > > > > > > > > > -- > > -- > > Post: opend...@googlegroups.com > > Unsubscribe: opendatakit...@googlegroups.com > > Options: http://groups.google.com/group/opendatakit?hl=en > > > > --- > > You received this message because you are subscribed to the Google Groups "ODK Community" group. > > To unsubscribe from this group and stop receiving emails from it, send an email to opendatakit...@googlegroups.com. > > For more options, visit https://groups.google.com/d/optout. > > > > > -- > > Mitch Sundt > Software Engineer > University of Washington > mitche...@gmail.com

Correct. The request was issued by ODK Aggregate and was killed by Google
after 60 seconds.

If Google infrastructure was working, it would have returned a value
(success or failure).

··· On Tue, May 24, 2016 at 3:02 PM, wrote:

Thanks Mitch,

So you're saying that the error I see in Aggregate:

Error: Problem persisting data or accessing data
([ChoiceSubmissionType.getValueFromEntity] SQL: SELECT * FROM
opendatakit.F6_V14_ECD4A WHERE _PARENT_AURI =
uuid:f6f98bdc-f55a-437a-9c3e-54718828ce3a ORDER BY _PARENT_AURI exception:
This request (380af43f8586449b) started at 2016/05/24 20:42:38.199 UTC and
was still executing at 2016/05/24 20:43:38.520 UTC.)

is due to an issue with Google infrastructure and therefore not something
that I can directly address (other than issuing a ticket)?

On Tuesday, May 24, 2016 at 5:30:45 PM UTC-4, Mitch Sundt wrote:

"Task was cancelled" indicates a problem in the Google infrastructure.
It does not necessarily mean that your datastore has a problem with this
record (Google is stuck and not returning success or failure).

You can open an issue with Google to find and fix this problem (this may
only be available to accounts with billing enabled), or you can wait. At
some point, they will likely fix it. I am unfortunately pessimistic about
their responsiveness to problem tickets (in your ticket, give them the GQL
you are trying to run).

If you were inserting a data record when Google began having trouble, it
is likely that once Google fixes the issue, you will then need to follow
the steps to fix the resulting data corruption, which may or may not be
related to this record.

On Tue, May 24, 2016 at 1:49 PM, pith....@gmail.com wrote:
Hi all,

I have encountered the above issue and have been trying to get it
resolved.

Error: Problem persisting data or accessing data
([ChoiceSubmissionType.getValueFromEntity] SQL: SELECT * FROM
opendatakit.F6_V14_ECD4A WHERE _PARENT_AURI =
uuid:f6f98bdc-f55a-437a-9c3e-54718828ce3a ORDER BY _PARENT_AURI exception:
This request (380af43f8586449b) started at 2016/05/24 20:42:38.199 UTC and
was still executing at 2016/05/24 20:43:38.520 UTC.)

I can confirm that if I lower the "Submissions per page" to eg 10, I no
longer encounter the error.

However, I have tried the steps outlined in:
https://github.com/opendatakit/opendatakit/wiki/Aggregate-AppEngine-Troubleshooting#reparing-a-filled-in-form-submission

without success.

From the error logs:

org.apache.commons.logging.impl.Jdk14Logger log: Retrying fetch with a
smaller chunk size: [ChoiceSubmissionType.getValueFromEntity] SQL: SELECT *
FROM opendatakit.F6_V14_ECD4A WHERE _PARENT_AURI =
uuid:f6f98bdc-f55a-437a-9c3e-54718828ce3a ORDER BY _PARENT_AURI exception:
Task was cancelled. (Jdk14Logger.java:87)

When I run the Query by GQL:

SELECT * FROM opendatakit.F6_V14_ECD4A WHERE _PARENT_AURI =
"uuid:f6f98bdc-f55a-437a-9c3e-54718828ce3a"

I get no results returned.

To confirm, I ran

SELECT * FROM opendatakit.F6_V14_ECD4A

And manually scanned for the uuid, couldn't find it

I'm not clear on why it would be querying for a UUID that doesn't exist,
but it's concerning. Any thoughts on how I should proceed?

Thanks in advance

--

--

Post: opend...@googlegroups.com

Unsubscribe: opendatakit...@googlegroups.com

Options: http://groups.google.com/group/opendatakit?hl=en


You received this message because you are subscribed to the Google
Groups "ODK Community" group.

To unsubscribe from this group and stop receiving emails from it, send
an email to opendatakit...@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--

Mitch Sundt
Software Engineer
University of Washington
mitche...@gmail.com

Thanks

--

Post: opendatakit@googlegroups.com
Unsubscribe: opendatakit+unsubscribe@googlegroups.com
Options: http://groups.google.com/group/opendatakit?hl=en


You received this message because you are subscribed to the Google Groups
"ODK Community" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to opendatakit+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--
Mitch Sundt
Software Engineer
University of Washington
mitchellsundt@gmail.com

To repair this, see

If you have trouble, let me know.

I have seen a flurry of cases of this in the last month.

··· On Sat, Jul 18, 2015 at 3:10 PM, wrote:

Thank you Lloyd and Mitch for getting to the bottom of this. We're having
the same issue and are relieved to hear that this error message doesn't
indicate any loss of data!

On Wednesday, March 25, 2015 at 12:42:07 AM UTC+9, Lloyd Owen Banwart wrote:

Re-posting an offline reply from Mitch here (I took the conversation
offline as I granted him developer access) that brings this issue to a
close.

"OK. Looking at the logs, I think everything is working as designed.

The form being displayed on the Submissions tab takes more than 60
seconds to completely fetch 100 rows from the database.

AppEngine imposes a 60-second limit, after which it aggressively
terminates any outstanding database calls, and the webserver detects and
returns that error up to the webpage. Issuing those queries directly in
the management console (
https://console.developers.google.com/project/tangointernational-3/datastore/query
) works fine and the results look like they should.

If you change the "Submissions per Page" to something less (e.g., 50),
the error should go away and you should be able to see the data in the
submissions tab.

If memory serves, that setting choice is not remembered once you leave
the page, so you would always have to re-enter it."

I can confirm that reducing the submissions per page seems to negate
this error.

Thank you Mitch for to diligent follow-up and assistance.

On Tuesday, March 24, 2015 at 10:58:28 AM UTC-4, Lloyd Owen Banwart wrote:
ODK Briefcase is working without failures. I do have paid instances.

On Tuesday, March 24, 2015 at 10:45:49 AM UTC-4, Mitch Sundt wrote:

And ODK Briefcase works to download the data, or does it give a failure?

If Briefcase is working OK without failures, and if you have a paid
instance, I would contact Google support, as this is a repeatable issue.
You should probably CC me on the support request you open with them, and
grant me developer access to the instance (and let me know the instance
URL) https://code.google.com/p/opendatakit/wiki/AppEngineTroubleshooting

On Mon, Mar 23, 2015 at 7:20 AM, Lloyd Owen Banwart lloyd....@gmail.com wrote:

To follow upon this post.

I had decreased access to the server for 24 hours to two users, and this
did not resolve the issue of receiving the below error. I then steadily
increased the frontend instance Class from 256 to 512 to 1024. In each
case, the error still was displayed on the ODK server. This error only
displays when viewing the data on the "Submissions tab" and under "Filter
Submissions" (no filter is applied). The error is repetitive, meaning that
when I leave the browser open, the error will display, instances may (or
may not) be shown in the submissions tab, then the error will pop up again.
Each time the error displays the error appears identical other than the
uuid, specific Q it is referencing, and start and end time of request.
I've tracked most uuids that have shown in the errors, and cross referenced
this with data obtained after the error has been displayed. All instances
displayed during errors are in the data which have been downloaded, and the
instances appear unaffected.

When accessing the data (via the "Export" function or via ODK Briefcase)
there appears to be no data loss when check against field-records of
uploaded instances compared to instances obtained from the server. Other
than being a annoying, and limiting any quick review of recently submitted
data via the server, I don't see any major issues at the moment with the
error -- but would love to get to the bottom of it.

Second example of error:
Error: Problem persisting
data or accessing data ([ChoiceSubmissionType.getValueFromEntity] SQL:
SELECT *
FROM opendatakit.ASUBL_FINAL2_ELG_B_SEC2_SEC3_MM_G3_LABELM3_2 WHERE
_PARENT_AURI = uuid:e1ba7512-9c8e-47d1-8458-0ddc1935642b ORDER BY
_PARENT_AURI
exception: This request (f44144cff1ba1b4f) started at 2015/03/23
13:51:54.645
UTC and was still executing at 2015/03/23 13:52:54.505 UTC.)

On Monday, March 23, 2015 at 9:39:14 AM UTC-4, Lloyd Owen Banwart wrote:
Mitch. Thanks (as always) for the clear and timely feedback.

  1. I've limited access, but it was a total of 3 with access before (now
    it is a total of 2) to the server.

  2. We use ODK briefcase (as we avail of the Stata ODKmeta program to
    provide value/var labels and restructure data sets from repeat groups).

  3. Re: Increasing the machine size; this is one of the smaller data
    collections we have done. The survey is large, but we are collecting data
    from less than 2,000 households --- is there a reason this issue would
    arise now and it hasn't in the past? Regardless, I'm waiting to see if
    limiting access to the server addresses this issue, if it does not I will
    increase the machine size.

I'll post progress here in the event someone else faces this issue in
the future. Thanks again.

~Lloyd

On Friday, March 20, 2015 at 10:35:18 PM UTC-4, Mitch Sundt wrote:

Several things could be going on:

(1) if many users are all watching the ODK Aggregate webpages at the
same time, they may be competing for processing space and time. Restrict
access to the server and provide nightly reports or publish the data into
Google Fusion Tables and provide access to that.

(2) if you are using "Export to CSV" to produce reports, switch to using
ODK Briefcase to download data to your local computer and then export to
CSV from there. The Export functionality on ODK Aggregate holds all of the
dataset in memory when generating a report. This will not work beyond a
certain size dataset.

(3) increase the "Frontend Instance Class" (machine size) on the
AppEngine ( https://appengine.google.com/ ). Choose your application id,
then click on Application Settings on the left sidebar. And select a larger
Frontend Instance Class.

Note that increasing the machine size will increase your costs.

On Fri, Mar 20, 2015 at 10:31 AM, Lloyd Owen Banwart < lloyd....@gmail.com> wrote:

Hi all,

We have a survey going live at the moment, with data being uploaded on a
daily basis. We are using a server a google app engine deployment of ODK
Aggregate version 1.4.3. It was my intention to update this to the latest
version of Aggregate once this fieldwork was completed.

To date, there has been no issues, but starting today when viewing data
on the "Submission" page of our appspot page for this server, I have been
getting the below error. The error is repetitive, but each time the uuid
case number changes.

"Error: Problem persisting data or accessing
data([ChoiceSubmissionType.getValueFromEntity] SQL: SELECT * FROM
opendatakit.ASUBL_FINAL2_ELG_B_SEC2_MJ_G6_J10C WHERE _PARENT_AURI =
uuid:6219f226-5a47-4501-b439-061dfce52d34 ORDER BY _PARENT_AURI
exception: This
request (2d170ed258bfabee) started at 2015/03/20 16:56:04.949 UTC and
was still
executing at 2015/03/20 16:57:04.779 UTC.)"

In the app engine application, when looking at the log, there are a
number of Info messages but thy all vary, and one warning message (see
below):

W2015-03-20 10:17:58.727 A problem was encountered with the process that
handled this request, causing it to exit. This is likely to cause a new
process to be used for the nex

Any advice on how to address this error, and if it will cause a
potential loss of data (all data has been backed up) in the future?

Thanks,

~Lloyd

--

--

Post: opend...@googlegroups.com

Unsubscribe: opendatakit...@googlegroups.com

Options: http://groups.google.com/group/opendatakit?hl=en


You received this message because you are subscribed to the Google
Groups "ODK Community" group.

To unsubscribe from this group and stop receiving emails from it, send
an email to opendatakit...@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--

Mitch Sundt
Software Engineer
University of Washington
mitche...@gmail.com

--

--

Post: opend...@googlegroups.com

Unsubscribe: opendatakit...@googlegroups.com

Options: http://groups.google.com/group/opendatakit?hl=en


You received this message because you are subscribed to the Google
Groups "ODK Community" group.

To unsubscribe from this group and stop receiving emails from it, send
an email to opendatakit...@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--

Mitch Sundt
Software Engineer
University of Washington
mitche...@gmail.com

--

Post: opendatakit@googlegroups.com
Unsubscribe: opendatakit+unsubscribe@googlegroups.com
Options: http://groups.google.com/group/opendatakit?hl=en


You received this message because you are subscribed to the Google Groups
"ODK Community" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to opendatakit+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--
Mitch Sundt
Software Engineer
University of Washington
mitchellsundt@gmail.com

Oof, ok, thanks Mitch

··· On Tuesday, May 24, 2016 at 6:59:56 PM UTC-4, Mitch Sundt wrote: > Correct. The request was issued by ODK Aggregate and was killed by Google after 60 seconds. > > > If Google infrastructure was working, it would have returned a value (success or failure). > > > > > On Tue, May 24, 2016 at 3:02 PM, wrote: > Thanks Mitch, > > > > So you're saying that the error I see in Aggregate: > > > > Error: Problem persisting data or accessing data ([ChoiceSubmissionType.getValueFromEntity] SQL: SELECT * FROM opendatakit.F6_V14_ECD4A WHERE _PARENT_AURI = uuid:f6f98bdc-f55a-437a-9c3e-54718828ce3a ORDER BY _PARENT_AURI exception: This request (380af43f8586449b) started at 2016/05/24 20:42:38.199 UTC and was still executing at 2016/05/24 20:43:38.520 UTC.) > > > > is due to an issue with Google infrastructure and therefore not something that I can directly address (other than issuing a ticket)? > > > > > > > > On Tuesday, May 24, 2016 at 5:30:45 PM UTC-4, Mitch Sundt wrote: > > > "Task was cancelled" indicates a problem in the Google infrastructure. It does not necessarily mean that your datastore has a problem with this record (Google is stuck and not returning success or failure). > > > > > > > > > You can open an issue with Google to find and fix this problem (this may only be available to accounts with billing enabled), or you can wait. At some point, they will likely fix it. I am unfortunately pessimistic about their responsiveness to problem tickets (in your ticket, give them the GQL you are trying to run). > > > > > > > > > > > > If you were inserting a data record when Google began having trouble, it is likely that once Google fixes the issue, you will then need to follow the steps to fix the resulting data corruption, which may or may not be related to this record. > > > > > > > > > > > > > > > On Tue, May 24, 2016 at 1:49 PM, wrote: > > > Hi all, > > > > > > > > > > > > I have encountered the above issue and have been trying to get it resolved. > > > > > > > > > > > > Error: Problem persisting data or accessing data ([ChoiceSubmissionType.getValueFromEntity] SQL: SELECT * FROM opendatakit.F6_V14_ECD4A WHERE _PARENT_AURI = uuid:f6f98bdc-f55a-437a-9c3e-54718828ce3a ORDER BY _PARENT_AURI exception: This request (380af43f8586449b) started at 2016/05/24 20:42:38.199 UTC and was still executing at 2016/05/24 20:43:38.520 UTC.) > > > > > > > > > > > > > > > > > > > > > > > > I can confirm that if I lower the "Submissions per page" to eg 10, I no longer encounter the error. > > > > > > > > > > > > However, I have tried the steps outlined in: https://github.com/opendatakit/opendatakit/wiki/Aggregate-AppEngine-Troubleshooting#reparing-a-filled-in-form-submission > > > > > > without success. > > > > > > > > > > > > From the error logs: > > > > > > org.apache.commons.logging.impl.Jdk14Logger log: Retrying fetch with a smaller chunk size: [ChoiceSubmissionType.getValueFromEntity] SQL: SELECT * FROM opendatakit.F6_V14_ECD4A WHERE _PARENT_AURI = uuid:f6f98bdc-f55a-437a-9c3e-54718828ce3a ORDER BY _PARENT_AURI exception: Task was cancelled. (Jdk14Logger.java:87) > > > > > > > > > > > > When I run the Query by GQL: > > > > > > SELECT * FROM `opendatakit.F6_V14_ECD4A` WHERE _PARENT_AURI = "uuid:f6f98bdc-f55a-437a-9c3e-54718828ce3a" > > > > > > I get no results returned. > > > > > > To confirm, I ran > > > > > > SELECT * FROM `opendatakit.F6_V14_ECD4A` > > > > > > And manually scanned for the uuid, couldn't find it > > > > > > > > > > > > > > > > > > I'm not clear on why it would be querying for a UUID that doesn't exist, but it's concerning. Any thoughts on how I should proceed? > > > > > > > > > > > > Thanks in advance > > > > > > > > > > > > > > > > > > -- > > > > > > -- > > > > > > Post: opend...@googlegroups.com > > > > > > Unsubscribe: opendatakit...@googlegroups.com > > > > > > Options: http://groups.google.com/group/opendatakit?hl=en > > > > > > > > > > > > --- > > > > > > You received this message because you are subscribed to the Google Groups "ODK Community" group. > > > > > > To unsubscribe from this group and stop receiving emails from it, send an email to opendatakit...@googlegroups.com. > > > > > > For more options, visit https://groups.google.com/d/optout. > > > > > > > > > > > > > > > > > > -- > > > > > > Mitch Sundt > > > Software Engineer > > > University of Washington > > > mitche...@gmail.com > > > > Thanks > > > > > > -- > > -- > > Post: opend...@googlegroups.com > > Unsubscribe: opendatakit...@googlegroups.com > > Options: http://groups.google.com/group/opendatakit?hl=en > > > > --- > > You received this message because you are subscribed to the Google Groups "ODK Community" group. > > To unsubscribe from this group and stop receiving emails from it, send an email to opendatakit...@googlegroups.com. > > For more options, visit https://groups.google.com/d/optout. > > > > > > -- > > Mitch Sundt > Software Engineer > University of Washington > mitche...@gmail.com