ODK on Google App Engine keeps running up charges

Greg · August 1, 2013, 7:35pm

Hi,
I'm running ODK Aggregate v.1.3.2 on Google App Engine and the
frontend-queue is running 24/7 with tasks and racking up some big charges.
Most of this time we are not using Aggregate. Google Enterprise Support
confirmed that most of the load came from /gae/uploadSubmissionsTask &
/gae/watchdog but claims everything is running normally on their end.

I'm dug through the server logs and most of the Info type of entries

are the

I 2013-08-01 13:02:30.447 [s~edmpestlab/1.369201346591658058].:
Time left on lock: -12380028.

 These Warning messages I hadn't seen before today:

W 2013-08-01 13:02:50.717
org.opendatakit.aggregate.task.UploadSubmissionsWorkerImpl
uploadAllSubmissions: Upload invoked when operational status is not ACTIVE

W 2013-08-01 13:02:50.953
org.opendatakit.aggregate.task.UploadSubmissionsWorkerImpl
uploadAllSubmissions: Unable to acquire lock

Examples of the error messages:

E 2013-08-01 13:03:08.889
org.opendatakit.aggregate.externalservice.GoogleSpreadsheet insertData:
Unable to insert data into spreadsheet
exception: OK

E 2013-08-01 13:03:09.236
org.opendatakit.aggregate.task.gae.servlet.UploadSubmissionsTaskServlet
doGet: org.opendatakit.aggregate.exception.ODKExternalServiceException:
com.go

E 2013-08-01 13:01:54.411
org.opendatakit.aggregate.task.gae.servlet.UploadSubmissionsTaskServlet
doGet: org.opendatakit.aggregate.exception.ODKExternalServiceException: {
"err

E 2013-08-01 13:03:07.456
org.opendatakit.aggregate.task.gae.servlet.UploadSubmissionsTaskServlet
doGet: org.opendatakit.aggregate.exception.ODKExternalServiceException: {
"error": { "errors": [ { "domain": "global", "reason":
"backendError", "message": "Backend Error" } ], "code": 503,
"message": "Backend Error" }}INSERT INTO
1IQS48A9Tq-R8s6geNV3yQvH4MPbaJCqsrX1861I
('meta-instance-id','meta-model-version','meta-ui-version','meta-submission-date','meta-is-complete','meta-date-marked-as-complete','FormVersion','DateAuto','deviceid','simid','PhoneNumber','Crew','PhoneUserLab','PhoneUserOps','PhoneUserOth','PhoneUser','Phonename')
VALUES
('uuid:ef0b5fd9-2ebd-487c-b451-dfeda5dd13a6','2013060402','','06/26/2013
16:04:44.807','true','06/26/2013
16:04:44.807','ver_2013-Jun-04_rev02','06/26/2013
16:04:09.785','356854040053167','8912230000040758245','','PestLab','BillSmith','','','BillSmith','Myphone
bill smith') ; INSERT INTO 1IQS48A9Tq-R8s6geNV3yQvH4MPbaJCqsrX1861I
('meta-instance-id','meta-model-version','meta-ui-version','meta-submission-date','meta-is-complete','meta-date-marked-as-complete','FormVersion','DateAuto','deviceid','simid','PhoneNumber','Crew','PhoneUserLab','PhoneUserOps','PhoneUserOth','PhoneUser','Phonename')
VALUES
('uuid:1db681c8-2716-4d7c-829a-45195b45955c','2013060402','','08/01/2013
18:41:53.837','true','08/01/2013
18:41:53.837','ver_2013-Jun-04_rev02','08/01/2013
18:41:34.003','356854040053100','8912230000127052512','17809185555','PestLab','JohnSmith','','','JohnSmith','Mine')

 The last error message in the list is the shortest example of an error

that happens for more than one form. I compared the data in these messages
with existing data in the corresponding Fusion Tables and it's always for
records that I had to directly alter in the Fusion Tables.

 Purging the waiting tasks in the frontend-queue stops activity in the

Dashboard charts briefly and pausing the frontend-queue stops most activity
for as long as it is paused while still queuing tasks. 256 tasks piled up
last night. I was running version 1.3.1 and then upgraded to 1.3.2 in
hopes that might solve the problem but nothing changed.

 To me it seems like something is "stuck" and keeps resubmitting but I

really don't know how to fix it. I'm a little cautious of pushing too many
buttons without knowing what things like "Flushing Memcache" etc. actually
do. I've searched through ODK Community and Google App Engine groups but
am not finding anything similar.

 Any suggestions of what to do, information needed to solve the

problem, places to refer my question, wild-ass guesses would be greatly
appreciated. As a side question, does anyone know of good documentation on
App Engine from an administrators point of view? All the documentation
I've seen seems oriented to programming.

Thank you.

Mitch_S · August 1, 2013, 10:13pm

As a stop-gap,

log onto ODK Aggregate
Go to Site Admin / Preferences
Check the checkbox labelled *'Disable faster background actions (exports,
publishing, form deletion) (slows quota usage on Google AppEngine)'

···

* This should restore your usage to prior values (it disables faster publishing, restoring behavior to the pre-1.3.1 behavior).

Do you have any validation rules set up on your fusion tables or
spreadsheets?
In the last example, 'Backend error' would indicate that it is an error on
the Fusion Tables side. Have you contacted the Fusion Tables team?

Do you have any validation rules set up on your tables or spreadsheets?

On Thu, Aug 1, 2013 at 12:35 PM, Greg greg.pommen@edmonton.ca wrote:

Hi,
I'm running ODK Aggregate v.1.3.2 on Google App Engine and the
frontend-queue is running 24/7 with tasks and racking up some big charges.
Most of this time we are not using Aggregate. Google Enterprise Support
confirmed that most of the load came from /gae/uploadSubmissionsTask &
/gae/watchdog but claims everything is running normally on their end.
I'm dug through the server logs and most of the Info type of entries
are the

I 2013-08-01 13:02:30.447 [s~edmpestlab/1.369201346591658058].:
Time left on lock: -12380028.
 These Warning messages I hadn't seen before today:
W 2013-08-01 13:02:50.717
org.opendatakit.aggregate.task.UploadSubmissionsWorkerImpl
uploadAllSubmissions: Upload invoked when operational status is not ACTIVE

W 2013-08-01 13:02:50.953
org.opendatakit.aggregate.task.UploadSubmissionsWorkerImpl
uploadAllSubmissions: Unable to acquire lock
Examples of the error messages:
E 2013-08-01 13:03:08.889
org.opendatakit.aggregate.externalservice.GoogleSpreadsheet insertData:
Unable to insert data into spreadsheet
exception: OK

E 2013-08-01 13:03:09.236
org.opendatakit.aggregate.task.gae.servlet.UploadSubmissionsTaskServlet
doGet: org.opendatakit.aggregate.exception.ODKExternalServiceException:
com.go

E 2013-08-01 13:01:54.411
org.opendatakit.aggregate.task.gae.servlet.UploadSubmissionsTaskServlet
doGet: org.opendatakit.aggregate.exception.ODKExternalServiceException: {
"err

E 2013-08-01 13:03:07.456
org.opendatakit.aggregate.task.gae.servlet.UploadSubmissionsTaskServlet
doGet: org.opendatakit.aggregate.exception.ODKExternalServiceException: {
"error": { "errors": [ { "domain": "global", "reason":
"backendError", "message": "Backend Error" } ], "code": 503,
"message": "Backend Error" }}INSERT INTO
1IQS48A9Tq-R8s6geNV3yQvH4MPbaJCqsrX1861I
('meta-instance-id','meta-model-version','meta-ui-version','meta-submission-date','meta-is-complete','meta-date-marked-as-complete','FormVersion','DateAuto','deviceid','simid','PhoneNumber','Crew','PhoneUserLab','PhoneUserOps','PhoneUserOth','PhoneUser','Phonename')
VALUES
('uuid:ef0b5fd9-2ebd-487c-b451-dfeda5dd13a6','2013060402','','06/26/2013
16:04:44.807','true','06/26/2013
16:04:44.807','ver_2013-Jun-04_rev02','06/26/2013
16:04:09.785','356854040053167','8912230000040758245','','PestLab','BillSmith','','','BillSmith','Myphone
bill smith') ; INSERT INTO 1IQS48A9Tq-R8s6geNV3yQvH4MPbaJCqsrX1861I
('meta-instance-id','meta-model-version','meta-ui-version','meta-submission-date','meta-is-complete','meta-date-marked-as-complete','FormVersion','DateAuto','deviceid','simid','PhoneNumber','Crew','PhoneUserLab','PhoneUserOps','PhoneUserOth','PhoneUser','Phonename')
VALUES
('uuid:1db681c8-2716-4d7c-829a-45195b45955c','2013060402','','08/01/2013
18:41:53.837','true','08/01/2013
18:41:53.837','ver_2013-Jun-04_rev02','08/01/2013
18:41:34.003','356854040053100','8912230000127052512','17809185555','PestLab','JohnSmith','','','JohnSmith','Mine')
 The last error message in the list is the shortest example of an
error that happens for more than one form. I compared the data in these
messages with existing data in the corresponding Fusion Tables and it's
always for records that I had to directly alter in the Fusion Tables.
 Purging the waiting tasks in the frontend-queue stops activity in the
Dashboard charts briefly and pausing the frontend-queue stops most activity
for as long as it is paused while still queuing tasks. 256 tasks piled up
last night. I was running version 1.3.1 and then upgraded to 1.3.2 in
hopes that might solve the problem but nothing changed.
 To me it seems like something is "stuck" and keeps resubmitting but I
really don't know how to fix it. I'm a little cautious of pushing too many
buttons without knowing what things like "Flushing Memcache" etc. actually
do. I've searched through ODK Community and Google App Engine groups but
am not finding anything similar.
 Any suggestions of what to do, information needed to solve the
problem, places to refer my question, wild-ass guesses would be greatly
appreciated. As a side question, does anyone know of good documentation on
App Engine from an administrators point of view? All the documentation
I've seen seems oriented to programming.

Thank you.

--

Post: opendatakit@googlegroups.com
Unsubscribe: opendatakit+unsubscribe@googlegroups.com
Options: http://groups.google.com/group/opendatakit?hl=en

You received this message because you are subscribed to the Google Groups
"ODK Community" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to opendatakit+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

--
Mitch Sundt
Software Engineer
University of Washington
mitchellsundt@gmail.com

Mitch_S · August 1, 2013, 10:16pm

Argh!

My untested suspicion is that you may have set up a validation rule or
constraint, and, when Aggregate tries to submit a record that violates that
constraint, Fusion Tables rejects the publishing of that submission,
causing the publisher to stall and never get past the failing record (it
sits in an infinite retry).

If that is what is happening, the two choices are:

(1) Log onto ODK Aggregate as a user with Forms Management privileges.
(2) find the offending records on the submissions page (e.g., via filtering)
(3) click the red 'X' to delete those records (permanently) from ODK
Aggregate

Or,

Eliminate the validation constraints in the Fusion Table or Spreadsheet
that you are publishing to.

Mitch

···

On Thu, Aug 1, 2013 at 3:13 PM, Mitch Sundt wrote:

As a stop-gap,

log onto ODK Aggregate
Go to Site Admin / Preferences
Check the checkbox labelled *'Disable faster background actions (exports,
publishing, form deletion) (slows quota usage on Google AppEngine)'

This should restore your usage to prior values (it disables faster
publishing, restoring behavior to the pre-1.3.1 behavior).

Do you have any validation rules set up on your fusion tables or
spreadsheets?
In the last example, 'Backend error' would indicate that it is an error on
the Fusion Tables side. Have you contacted the Fusion Tables team?

Do you have any validation rules set up on your tables or spreadsheets?

On Thu, Aug 1, 2013 at 12:35 PM, Greg greg.pommen@edmonton.ca wrote:
Hi,
I'm running ODK Aggregate v.1.3.2 on Google App Engine and the
frontend-queue is running 24/7 with tasks and racking up some big charges.
Most of this time we are not using Aggregate. Google Enterprise Support
confirmed that most of the load came from /gae/uploadSubmissionsTask &
/gae/watchdog but claims everything is running normally on their end.
I'm dug through the server logs and most of the Info type of entries
are the

I 2013-08-01 13:02:30.447 [s~edmpestlab/1.369201346591658058].:
Time left on lock: -12380028.
 These Warning messages I hadn't seen before today:
W 2013-08-01 13:02:50.717
org.opendatakit.aggregate.task.UploadSubmissionsWorkerImpl
uploadAllSubmissions: Upload invoked when operational status is not ACTIVE

W 2013-08-01 13:02:50.953
org.opendatakit.aggregate.task.UploadSubmissionsWorkerImpl
uploadAllSubmissions: Unable to acquire lock
Examples of the error messages:
E 2013-08-01 13:03:08.889
org.opendatakit.aggregate.externalservice.GoogleSpreadsheet insertData:
Unable to insert data into spreadsheet
exception: OK

E 2013-08-01 13:03:09.236
org.opendatakit.aggregate.task.gae.servlet.UploadSubmissionsTaskServlet
doGet: org.opendatakit.aggregate.exception.ODKExternalServiceException:
com.go

E 2013-08-01 13:01:54.411
org.opendatakit.aggregate.task.gae.servlet.UploadSubmissionsTaskServlet
doGet: org.opendatakit.aggregate.exception.ODKExternalServiceException: {
"err

E 2013-08-01 13:03:07.456
org.opendatakit.aggregate.task.gae.servlet.UploadSubmissionsTaskServlet
doGet: org.opendatakit.aggregate.exception.ODKExternalServiceException: {
"error": { "errors": [ { "domain": "global", "reason":
"backendError", "message": "Backend Error" } ], "code": 503,
"message": "Backend Error" }}INSERT INTO
1IQS48A9Tq-R8s6geNV3yQvH4MPbaJCqsrX1861I
('meta-instance-id','meta-model-version','meta-ui-version','meta-submission-date','meta-is-complete','meta-date-marked-as-complete','FormVersion','DateAuto','deviceid','simid','PhoneNumber','Crew','PhoneUserLab','PhoneUserOps','PhoneUserOth','PhoneUser','Phonename')
VALUES
('uuid:ef0b5fd9-2ebd-487c-b451-dfeda5dd13a6','2013060402','','06/26/2013
16:04:44.807','true','06/26/2013
16:04:44.807','ver_2013-Jun-04_rev02','06/26/2013
16:04:09.785','356854040053167','8912230000040758245','','PestLab','BillSmith','','','BillSmith','Myphone
bill smith') ; INSERT INTO 1IQS48A9Tq-R8s6geNV3yQvH4MPbaJCqsrX1861I
('meta-instance-id','meta-model-version','meta-ui-version','meta-submission-date','meta-is-complete','meta-date-marked-as-complete','FormVersion','DateAuto','deviceid','simid','PhoneNumber','Crew','PhoneUserLab','PhoneUserOps','PhoneUserOth','PhoneUser','Phonename')
VALUES
('uuid:1db681c8-2716-4d7c-829a-45195b45955c','2013060402','','08/01/2013
18:41:53.837','true','08/01/2013
18:41:53.837','ver_2013-Jun-04_rev02','08/01/2013
18:41:34.003','356854040053100','8912230000127052512','17809185555','PestLab','JohnSmith','','','JohnSmith','Mine')
 The last error message in the list is the shortest example of an
error that happens for more than one form. I compared the data in these
messages with existing data in the corresponding Fusion Tables and it's
always for records that I had to directly alter in the Fusion Tables.
 Purging the waiting tasks in the frontend-queue stops activity in
the Dashboard charts briefly and pausing the frontend-queue stops most
activity for as long as it is paused while still queuing tasks. 256 tasks
piled up last night. I was running version 1.3.1 and then upgraded to
1.3.2 in hopes that might solve the problem but nothing changed.
 To me it seems like something is "stuck" and keeps resubmitting but
I really don't know how to fix it. I'm a little cautious of pushing too
many buttons without knowing what things like "Flushing Memcache" etc.
actually do. I've searched through ODK Community and Google App Engine
groups but am not finding anything similar.
 Any suggestions of what to do, information needed to solve the
problem, places to refer my question, wild-ass guesses would be greatly
appreciated. As a side question, does anyone know of good documentation on
App Engine from an administrators point of view? All the documentation
I've seen seems oriented to programming.

Thank you.

--

Post: opendatakit@googlegroups.com
Unsubscribe: opendatakit+unsubscribe@googlegroups.com
Options: http://groups.google.com/group/opendatakit?hl=en

You received this message because you are subscribed to the Google Groups
"ODK Community" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to opendatakit+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.
--
Mitch Sundt
Software Engineer
University of Washington
mitchellsundt@gmail.com

--
Mitch Sundt
Software Engineer
University of Washington
mitchellsundt@gmail.com

Greg · August 2, 2013, 5:06pm

Hi Mitch,

 I checked the *'D**isable faster background actions (exports,

publishing, form deletion) (slows quota usage on Google AppEngine)'* and
that has helped immensely. The logs are still full of the ": Time
left on lock:" entries but the Errors have mostly disappeared and we aren't
getting a constant (and expensive) /gae/uploadSubmissionsTask load.

 I wasn't even aware of the validation rules in Fusion and

Spreadsheets. That function wasn't available from the "Change Variables"
section of Fusion Tables when I last set these up in June. I checked
through anyway and no validation rules are in place. However I do have
calculated columns in place. Like I said though, from the few records I
was able to glean from errors, the ones it keeps trying to insert and
getting errors on, are the same ones I've made manual changes to in Fusion
Tables. The spreadsheets also return errors but nothing to indicate which
entries are resulting in errors. Admittedly though I'm not the best at
sorting through the logs and mostly I just keep looking through until my
eyeballs bleed. I don't seem to understand the flavour of regex used.

So as for deleting offending records, perhaps I can just delete all the

records of the offending tables since we use Fusion Tables for our data.
As for the "Disable faster background actions", what is the long term
consequence of keeping that box checked? I feel a little nervous about
keeping a stop gap measure in place without curing the initial source of
the problem. I'll have to contact the Fusion Tables team too but I wonder
if they will be like the Google App Engine team that basically said
Google's end is good, it's really my problem.

 I've added you as a developer to App Engine in case you want to snoop

around and see what's wrong or research for ODK development. Thanks again
for the help!

···

On Thursday, August 1, 2013 4:16:44 PM UTC-6, Mitch Sundt wrote: > > Argh! > > My untested suspicion is that you may have set up a validation rule or > constraint, and, when Aggregate tries to submit a record that violates that > constraint, Fusion Tables rejects the publishing of that submission, > causing the publisher to stall and never get past the failing record (it > sits in an infinite retry). > > If that is what is happening, the two choices are: > > (1) Log onto ODK Aggregate as a user with Forms Management privileges. > (2) find the offending records on the submissions page (e.g., via > filtering) > (3) click the red 'X' to delete those records (permanently) from ODK > Aggregate > > Or, > > Eliminate the validation constraints in the Fusion Table or Spreadsheet > that you are publishing to. > > Mitch > > > > On Thu, Aug 1, 2013 at 3:13 PM, Mitch Sundt <mitche...@gmail.com wrote: > >> As a stop-gap, >> >> log onto ODK Aggregate >> Go to Site Admin / Preferences >> Check the checkbox labelled *'Disable faster background actions >> (exports, publishing, form deletion) (slows quota usage on Google >> AppEngine)' >> >> * >> This should restore your usage to prior values (it disables faster >> publishing, restoring behavior to the pre-1.3.1 behavior). >> >> Do you have any validation rules set up on your fusion tables or >> spreadsheets? >> In the last example, 'Backend error' would indicate that it is an error >> on the Fusion Tables side. Have you contacted the Fusion Tables team? >> >> Do you have any validation rules set up on your tables or spreadsheets? >> >> * >> * >> >> >> On Thu, Aug 1, 2013 at 12:35 PM, Greg <greg....@edmonton.ca wrote: >> >>> Hi, >>> I'm running ODK Aggregate v.1.3.2 on Google App Engine and the >>> frontend-queue is running 24/7 with tasks and racking up some big charges. >>> Most of this time we are not using Aggregate. Google Enterprise Support >>> confirmed that most of the load came from /gae/uploadSubmissionsTask & >>> /gae/watchdog but claims everything is running normally on their end. >>> >>> >>> I'm dug through the server logs and most of the Info type of entries >>> are the >>> >>> I 2013-08-01 13:02:30.447 [s~edmpestlab/1.369201346591658058].: >>> Time left on lock: -12380028. >>> >>> >>> These Warning messages I hadn't seen before today: >>> >>> W 2013-08-01 13:02:50.717 >>> org.opendatakit.aggregate.task.UploadSubmissionsWorkerImpl >>> uploadAllSubmissions: Upload invoked when operational status is not ACTIVE >>> >>> W 2013-08-01 13:02:50.953 >>> org.opendatakit.aggregate.task.UploadSubmissionsWorkerImpl >>> uploadAllSubmissions: Unable to acquire lock >>> >>> >>> Examples of the error messages: >>> >>> E 2013-08-01 13:03:08.889 >>> org.opendatakit.aggregate.externalservice.GoogleSpreadsheet insertData: >>> Unable to insert data into spreadsheet >>> exception: OK >>> >>> E 2013-08-01 13:03:09.236 >>> org.opendatakit.aggregate.task.gae.servlet.UploadSubmissionsTaskServlet >>> doGet: org.opendatakit.aggregate.exception.ODKExternalServiceException: >>> com.go >>> >>> E 2013-08-01 13:01:54.411 >>> org.opendatakit.aggregate.task.gae.servlet.UploadSubmissionsTaskServlet >>> doGet: org.opendatakit.aggregate.exception.ODKExternalServiceException: { >>> "err >>> >>> E 2013-08-01 13:03:07.456 >>> org.opendatakit.aggregate.task.gae.servlet.UploadSubmissionsTaskServlet >>> doGet: org.opendatakit.aggregate.exception.ODKExternalServiceException: { >>> "error": { "errors": [ { "domain": "global", "reason": >>> "backendError", "message": "Backend Error" } ], "code": 503, >>> "message": "Backend Error" }}INSERT INTO >>> 1IQS48A9Tq-R8s6geNV3yQvH4MPbaJCqsrX1861I >>> ('*meta-instance-id*','*meta-model-version*','*meta-ui-version*','*meta-submission-date*','*meta-is-complete*','*meta-date-marked-as-complete*','FormVersion','DateAuto','deviceid','simid','PhoneNumber','Crew','PhoneUserLab','PhoneUserOps','PhoneUserOth','PhoneUser','Phonename') >>> VALUES >>> ('uuid:ef0b5fd9-2ebd-487c-b451-dfeda5dd13a6','2013060402','','06/26/2013 >>> 16:04:44.807','true','06/26/2013 >>> 16:04:44.807','ver_2013-Jun-04_rev02','06/26/2013 >>> 16:04:09.785','356854040053167','8912230000040758245','','PestLab','BillSmith','','','BillSmith','Myphone >>> bill smith') ; INSERT INTO 1IQS48A9Tq-R8s6geNV3yQvH4MPbaJCqsrX1861I >>> ('*meta-instance-id*','*meta-model-version*','*meta-ui-version*','*meta-submission-date*','*meta-is-complete*','*meta-date-marked-as-complete*','FormVersion','DateAuto','deviceid','simid','PhoneNumber','Crew','PhoneUserLab','PhoneUserOps','PhoneUserOth','PhoneUser','Phonename') >>> VALUES >>> ('uuid:1db681c8-2716-4d7c-829a-45195b45955c','2013060402','','08/01/2013 >>> 18:41:53.837','true','08/01/2013 >>> 18:41:53.837','ver_2013-Jun-04_rev02','08/01/2013 >>> 18:41:34.003','356854040053100','8912230000127052512','17809185555','PestLab','JohnSmith','','','JohnSmith','Mine') >>> >>> >>> >>> The last error message in the list is the shortest example of an >>> error that happens for more than one form. I compared the data in these >>> messages with existing data in the corresponding Fusion Tables and it's >>> always for records that I had to directly alter in the Fusion Tables. >>> >>> Purging the waiting tasks in the frontend-queue stops activity in >>> the Dashboard charts briefly and pausing the frontend-queue stops most >>> activity for as long as it is paused while still queuing tasks. 256 tasks >>> piled up last night. I was running version 1.3.1 and then upgraded to >>> 1.3.2 in hopes that might solve the problem but nothing changed. >>> >>> To me it seems like something is "stuck" and keeps resubmitting but >>> I really don't know how to fix it. I'm a little cautious of pushing too >>> many buttons without knowing what things like "Flushing Memcache" etc. >>> actually do. I've searched through ODK Community and Google App Engine >>> groups but am not finding anything similar. >>> >>> Any suggestions of what to do, information needed to solve the >>> problem, places to refer my question, wild-ass guesses would be greatly >>> appreciated. As a side question, does anyone know of good documentation on >>> App Engine from an administrators point of view? All the documentation >>> I've seen seems oriented to programming. >>> >>> Thank you. >>> >>> >>> >>> -- >>> -- >>> Post: opend...@googlegroups.com >>> Unsubscribe: opendatakit...@googlegroups.com >>> Options: http://groups.google.com/group/opendatakit?hl=en >>> >>> --- >>> You received this message because you are subscribed to the Google >>> Groups "ODK Community" group. >>> To unsubscribe from this group and stop receiving emails from it, send >>> an email to opendatakit...@googlegroups.com . >>> For more options, visit https://groups.google.com/groups/opt_out. >>> >>> >>> >> >> >> >> -- >> Mitch Sundt >> Software Engineer >> University of Washington >> mitche...@gmail.com >> > > > > -- > Mitch Sundt > Software Engineer > University of Washington > mitche...@gmail.com >

Mitch_S · August 2, 2013, 7:34pm

Not sure if you saw this e-mail I sent to the edmptestlab@ gmail account:
---start--------------------
Hi,

A number of your publishers are in the 'PAUSED' state which indicates
either a transient or a ongoing failure to publish to the destination
servers.

In one instance, for this publisher:

Form: Check-In Form
Establishment Date: 2013-06-05 21:26:24.969000
Mode: Both Uploading and Streaming

The error is that the fusion table no longer exists.

You should double-check that the links to the fusion tables and
spreadsheets that are in the 'PAUSED' state actually work. If they don't,
you should delete those publishers.

Otherwise, ODK Aggregate will keep attempting to publish to those
locations, fail, and re-attempt it a short time later. Right now, that is
every 90 seconds. In a future release we will retry once 90 seconds later,
then retry only every 7.5 minutes, or every 15 minutes if the server is not
in use.

--end--------------------

w.r.t. keeping the box checked -- this simply sets the behavior back to the
highly conservative pre-1.3.1 publishing rate (a much slower rate, aimed at
conserving quota for interactive use, and trickling data into Fusion
Tables, etc.). Once the 7.5 minute back-off is available (mentioned in
the above e-mail), you should not need to check this checkbox, as the
server will then automatically shift failing publishers into a
highly-conservative publishing-retry cycle, and move them back into the
aggressive publishing cycle once they stop failing.

Mitch

···

To: Fusion Tables

On Fri, Aug 2, 2013 at 10:06 AM, Greg greg.pommen@edmonton.ca wrote:

Hi Mitch,
 I checked the *'D**isable faster background actions (exports,
publishing, form deletion) (slows quota usage on Google AppEngine)'* and
that has helped immensely. The logs are still full of the ": Time
left on lock:" entries but the Errors have mostly disappeared and we aren't
getting a constant (and expensive) /gae/uploadSubmissionsTask load.
 I wasn't even aware of the validation rules in Fusion and
Spreadsheets. That function wasn't available from the "Change Variables"
section of Fusion Tables when I last set these up in June. I checked
through anyway and no validation rules are in place. However I do have
calculated columns in place. Like I said though, from the few records I
was able to glean from errors, the ones it keeps trying to insert and
getting errors on, are the same ones I've made manual changes to in Fusion
Tables. The spreadsheets also return errors but nothing to indicate which
entries are resulting in errors. Admittedly though I'm not the best at
sorting through the logs and mostly I just keep looking through until my
eyeballs bleed. I don't seem to understand the flavour of regex used.
So as for deleting offending records, perhaps I can just delete all
the records of the offending tables since we use Fusion Tables for our
data. As for the "Disable faster background actions", what is the long
term consequence of keeping that box checked? I feel a little nervous
about keeping a stop gap measure in place without curing the initial source
of the problem. I'll have to contact the Fusion Tables team too but I
wonder if they will be like the Google App Engine team that basically said
Google's end is good, it's really my problem.
 I've added you as a developer to App Engine in case you want to snoop
around and see what's wrong or research for ODK development. Thanks again
for the help!

On Thursday, August 1, 2013 4:16:44 PM UTC-6, Mitch Sundt wrote:
Argh!

My untested suspicion is that you may have set up a validation rule or
constraint, and, when Aggregate tries to submit a record that violates that
constraint, Fusion Tables rejects the publishing of that submission,
causing the publisher to stall and never get past the failing record (it
sits in an infinite retry).

If that is what is happening, the two choices are:

(1) Log onto ODK Aggregate as a user with Forms Management privileges.
(2) find the offending records on the submissions page (e.g., via
filtering)
(3) click the red 'X' to delete those records (permanently) from ODK
Aggregate

Or,

Eliminate the validation constraints in the Fusion Table or Spreadsheet
that you are publishing to.

Mitch

On Thu, Aug 1, 2013 at 3:13 PM, Mitch Sundt mitche...@gmail.com wrote:
As a stop-gap,

log onto ODK Aggregate
Go to Site Admin / Preferences
Check the checkbox labelled *'Disable faster background actions
(exports, publishing, form deletion) (slows quota usage on Google
AppEngine)'

This should restore your usage to prior values (it disables faster
publishing, restoring behavior to the pre-1.3.1 behavior).

Do you have any validation rules set up on your fusion tables or
spreadsheets?
In the last example, 'Backend error' would indicate that it is an error
on the Fusion Tables side. Have you contacted the Fusion Tables team?

Do you have any validation rules set up on your tables or spreadsheets?

On Thu, Aug 1, 2013 at 12:35 PM, Greg greg....@edmonton.ca wrote:
Hi,
I'm running ODK Aggregate v.1.3.2 on Google App Engine and the
frontend-queue is running 24/7 with tasks and racking up some big charges.
Most of this time we are not using Aggregate. Google Enterprise Support
confirmed that most of the load came from /gae/uploadSubmissionsTask &
/gae/watchdog but claims everything is running normally on their end.
I'm dug through the server logs and most of the Info type of
entries are the

I 2013-08-01 13:02:30.447 [s~edmpestlab/1.**369201346591658058].:
Time left on lock: -12380028.
 These Warning messages I hadn't seen before today:
W 2013-08-01 13:02:50.717 org.opendatakit.aggregate.**task.**UploadSubmissionsWorkerImpl
uploadAllSubmissions: Upload invoked when operational status is not ACTIVE

W 2013-08-01 13:02:50.953 org.opendatakit.aggregate.**task.**UploadSubmissionsWorkerImpl
uploadAllSubmissions: Unable to acquire lock
Examples of the error messages:
E 2013-08-01 13:03:08.889 org.opendatakit.aggregate.*externalservice.
*GoogleSpreadsheet insertData: Unable to insert data into spreadsheet
exception: OK

E 2013-08-01 13:03:09.236 org.opendatakit.aggregate.**task.gae.servlet.
UploadSubmissionsTaskServlet doGet: org.opendatakit.aggregate.
exception.**ODKExternalServiceException: com.go

E 2013-08-01 13:01:54.411 org.opendatakit.aggregate.**task.gae.servlet.
UploadSubmissionsTaskServlet doGet: org.opendatakit.aggregate.
exception.**ODKExternalServiceException: { "err

E 2013-08-01 13:03:07.456
org.opendatakit.aggregate.**task.gae.servlet.**UploadSubmissionsTaskServlet
doGet: org.opendatakit.aggregate.exception.ODKExternalServiceException:
{ "error": { "errors": [ { "domain": "global", "reason":
"backendError", "message": "Backend Error" } ], "code": 503,
"message": "Backend Error" }}INSERT INTO 1IQS48A9Tq-R8s6geNV3yQvH4MPbaJCqsrX1861I
('meta-instance-id','meta-model-version','meta-ui-
version','meta-submission-date','meta-is-complete','
meta-date-marked-as-complete','FormVersion','DateAuto','
deviceid','simid','PhoneNumber','Crew','
PhoneUserLab','PhoneUserOps','**PhoneUserOth','PhoneUser','**Phonename')
VALUES ('uuid:ef0b5fd9-2ebd-487c-**b451-dfeda5dd13a6','**2013060402','','06/26/2013
16:04:44.807','true','06/26/**2013 16:04:44.807','ver_2013-Jun-**04_rev02','06/26/2013
16:04:09.785','356854040053167','8912230000040758245','','
PestLab','BillSmith','','','BillSmith','Myphone bill smith') ;
INSERT INTO 1IQS48A9Tq-R8s6geNV3yQvH4MPbaJCqsrX1861I
('meta-instance-id','meta-model-version','meta-ui-
version','meta-submission-date','meta-is-complete','
meta-date-marked-as-complete','FormVersion','DateAuto','
deviceid','simid','PhoneNumber','Crew','
PhoneUserLab','PhoneUserOps','**PhoneUserOth','PhoneUser','**Phonename')
VALUES ('uuid:1db681c8-2716-4d7c-**829a-45195b45955c','**2013060402','','08/01/2013
18:41:53.837','true','08/01/**2013 18:41:53.837','ver_2013-Jun-**04_rev02','08/01/2013
18:41:34.003','**356854040053100','8912230000127052512','
17809185555','PestLab','**JohnSmith','','','JohnSmith','**Mine')
 The last error message in the list is the shortest example of an
error that happens for more than one form. I compared the data in these
messages with existing data in the corresponding Fusion Tables and it's
always for records that I had to directly alter in the Fusion Tables.
 Purging the waiting tasks in the frontend-queue stops activity in
the Dashboard charts briefly and pausing the frontend-queue stops most
activity for as long as it is paused while still queuing tasks. 256 tasks
piled up last night. I was running version 1.3.1 and then upgraded to
1.3.2 in hopes that might solve the problem but nothing changed.
 To me it seems like something is "stuck" and keeps resubmitting
but I really don't know how to fix it. I'm a little cautious of pushing
too many buttons without knowing what things like "Flushing Memcache" etc.
actually do. I've searched through ODK Community and Google App Engine
groups but am not finding anything similar.
 Any suggestions of what to do, information needed to solve the
problem, places to refer my question, wild-ass guesses would be greatly
appreciated. As a side question, does anyone know of good documentation on
App Engine from an administrators point of view? All the documentation
I've seen seems oriented to programming.

Thank you.

--

Post: opend...@googlegroups.com
Unsubscribe: opendatakit...@**googlegroups.com

Options: http://groups.google.com/**group/opendatakit?hl=en http://groups.google.com/group/opendatakit?hl=en

You received this message because you are subscribed to the Google
Groups "ODK Community" group.
To unsubscribe from this group and stop receiving emails from it, send
an email to opendatakit...@**googlegroups.com.

For more options, visit https://groups.google.com/**groups/opt_out https://groups.google.com/groups/opt_out
.
--
Mitch Sundt
Software Engineer
University of Washington
mitche...@gmail.com
--
Mitch Sundt
Software Engineer
University of Washington
mitche...@gmail.com
--

Post: opendatakit@googlegroups.com
Unsubscribe: opendatakit+unsubscribe@googlegroups.com
Options: http://groups.google.com/group/opendatakit?hl=en

You received this message because you are subscribed to the Google Groups
"ODK Community" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to opendatakit+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

--
Mitch Sundt
Software Engineer
University of Washington
mitchellsundt@gmail.com

Greg · August 2, 2013, 9:30pm

Hi,
No I didn't see that, it's a shared account so thank you very much for
letting me know. I'm just leaving for the weekend but I suspect you
probably found the issue. I didn't know ODK would retain publishing links
like that. I'm assuming it does that after a form upgrade? Or would that
be after deleting a form and uploading a new one of the same name and ID.
In any case I'll have to check when I changed the form and delete the
PAUSED ones.

Thank you very much again!

···

On Friday, August 2, 2013 1:34:08 PM UTC-6, Mitch Sundt wrote: > > Not sure if you saw this e-mail I sent to the edmptestlab@ gmail account: > ---start-------------------- > Hi, > > A number of your publishers are in the 'PAUSED' state which indicates > either a transient or a ongoing failure to publish to the destination > servers. > > In one instance, for this publisher: > > Form: Check-In Form > To: Fusion Tables > Establishment Date: 2013-06-05 21:26:24.969000 > Mode: Both Uploading and Streaming > > The error is that the fusion table no longer exists. > > You should double-check that the links to the fusion tables and > spreadsheets that are in the 'PAUSED' state actually work. If they don't, > you should delete those publishers. > > Otherwise, ODK Aggregate will keep attempting to publish to those > locations, fail, and re-attempt it a short time later. Right now, that is > every 90 seconds. In a future release we will retry once 90 seconds later, > then retry only every 7.5 minutes, or every 15 minutes if the server is not > in use. > > --end-------------------- > > w.r.t. keeping the box checked -- this simply sets the behavior back to > the highly conservative pre-1.3.1 publishing rate (a much slower rate, > aimed at conserving quota for interactive use, and trickling data into > Fusion Tables, etc.). Once the 7.5 minute back-off is available > (mentioned in the above e-mail), you should not need to check this > checkbox, as the server will then automatically shift failing publishers > into a highly-conservative publishing-retry cycle, and move them back into > the aggressive publishing cycle once they stop failing. > > Mitch > > On Fri, Aug 2, 2013 at 10:06 AM, Greg <greg....@edmonton.ca >wrote: > >> Hi Mitch, >> >> I checked the *'D**isable faster background actions (exports, >> publishing, form deletion) (slows quota usage on Google AppEngine)'* and >> that has helped immensely. The logs are still full of the ": Time >> left on lock:" entries but the Errors have mostly disappeared and we aren't >> getting a constant (and expensive) /gae/uploadSubmissionsTask load. >> >> I wasn't even aware of the validation rules in Fusion and >> Spreadsheets. That function wasn't available from the "Change Variables" >> section of Fusion Tables when I last set these up in June. I checked >> through anyway and no validation rules are in place. However I do have >> calculated columns in place. Like I said though, from the few records I >> was able to glean from errors, the ones it keeps trying to insert and >> getting errors on, are the same ones I've made manual changes to in Fusion >> Tables. The spreadsheets also return errors but nothing to indicate which >> entries are resulting in errors. Admittedly though I'm not the best at >> sorting through the logs and mostly I just keep looking through until my >> eyeballs bleed. I don't seem to understand the flavour of regex used. >> >> So as for deleting offending records, perhaps I can just delete all >> the records of the offending tables since we use Fusion Tables for our >> data. As for the "Disable faster background actions", what is the long >> term consequence of keeping that box checked? I feel a little nervous >> about keeping a stop gap measure in place without curing the initial source >> of the problem. I'll have to contact the Fusion Tables team too but I >> wonder if they will be like the Google App Engine team that basically said >> Google's end is good, it's really my problem. >> >> I've added you as a developer to App Engine in case you want to >> snoop around and see what's wrong or research for ODK development. Thanks >> again for the help! >> >> >> >> >> >> On Thursday, August 1, 2013 4:16:44 PM UTC-6, Mitch Sundt wrote: >> >>> Argh! >>> >>> My untested suspicion is that you may have set up a validation rule or >>> constraint, and, when Aggregate tries to submit a record that violates that >>> constraint, Fusion Tables rejects the publishing of that submission, >>> causing the publisher to stall and never get past the failing record (it >>> sits in an infinite retry). >>> >>> If that is what is happening, the two choices are: >>> >>> (1) Log onto ODK Aggregate as a user with Forms Management privileges. >>> (2) find the offending records on the submissions page (e.g., via >>> filtering) >>> (3) click the red 'X' to delete those records (permanently) from ODK >>> Aggregate >>> >>> Or, >>> >>> Eliminate the validation constraints in the Fusion Table or Spreadsheet >>> that you are publishing to. >>> >>> Mitch >>> >>> >>> >>> On Thu, Aug 1, 2013 at 3:13 PM, Mitch Sundt wrote: >>> >>>> As a stop-gap, >>>> >>>> log onto ODK Aggregate >>>> Go to Site Admin / Preferences >>>> Check the checkbox labelled *'Disable faster background actions >>>> (exports, publishing, form deletion) (slows quota usage on Google >>>> AppEngine)' >>>> >>>> * >>>> This should restore your usage to prior values (it disables faster >>>> publishing, restoring behavior to the pre-1.3.1 behavior). >>>> >>>> Do you have any validation rules set up on your fusion tables or >>>> spreadsheets? >>>> In the last example, 'Backend error' would indicate that it is an error >>>> on the Fusion Tables side. Have you contacted the Fusion Tables team? >>>> >>>> Do you have any validation rules set up on your tables or spreadsheets? >>>> >>>> * >>>> * >>>> >>>> >>>> On Thu, Aug 1, 2013 at 12:35 PM, Greg wrote: >>>> >>>>> Hi, >>>>> I'm running ODK Aggregate v.1.3.2 on Google App Engine and the >>>>> frontend-queue is running 24/7 with tasks and racking up some big charges. >>>>> Most of this time we are not using Aggregate. Google Enterprise Support >>>>> confirmed that most of the load came from /gae/uploadSubmissionsTask & >>>>> /gae/watchdog but claims everything is running normally on their end. >>>>> >>>>> >>>>> I'm dug through the server logs and most of the Info type of >>>>> entries are the >>>>> >>>>> I 2013-08-01 13:02:30.447 [s~edmpestlab/1.**369201346591658058].: >>>>> Time left on lock: -12380028. >>>>> >>>>> >>>>> These Warning messages I hadn't seen before today: >>>>> >>>>> W 2013-08-01 13:02:50.717 org.opendatakit.aggregate.**task.**UploadSubmissionsWorkerImpl >>>>> uploadAllSubmissions: Upload invoked when operational status is not ACTIVE >>>>> >>>>> W 2013-08-01 13:02:50.953 org.opendatakit.aggregate.**task.**UploadSubmissionsWorkerImpl >>>>> uploadAllSubmissions: Unable to acquire lock >>>>> >>>>> >>>>> Examples of the error messages: >>>>> >>>>> E 2013-08-01 13:03:08.889 org.opendatakit.aggregate.**externalservice. >>>>> **GoogleSpreadsheet insertData: Unable to insert data into >>>>> spreadsheet exception: OK >>>>> >>>>> E 2013-08-01 13:03:09.236 org.opendatakit.aggregate.** >>>>> task.gae.servlet.**UploadSubmissionsTaskServlet doGet: >>>>> org.opendatakit.aggregate.**exception.**ODKExternalServiceException: >>>>> com.go >>>>> >>>>> E 2013-08-01 13:01:54.411 org.opendatakit.aggregate.** >>>>> task.gae.servlet.**UploadSubmissionsTaskServlet doGet: >>>>> org.opendatakit.aggregate.**exception.**ODKExternalServiceException: >>>>> { "err >>>>> >>>>> E 2013-08-01 13:03:07.456 >>>>> org.opendatakit.aggregate.**task.gae.servlet.**UploadSubmissionsTaskServlet >>>>> doGet: org.opendatakit.aggregate.**exception.**ODKExternalServiceException: >>>>> { "error": { "errors": [ { "domain": "global", "reason": >>>>> "backendError", "message": "Backend Error" } ], "code": 503, >>>>> "message": "Backend Error" }}INSERT INTO 1IQS48A9Tq-**R8s6geNV3yQvH4MPbaJCqsrX1861I >>>>> ('*meta-instance-id*','*meta-**model-version*','*meta-ui-** >>>>> version*','*meta-submission-**date*','*meta-is-complete*','*** >>>>> meta-date-marked-as-complete*'**,'FormVersion','DateAuto','** >>>>> deviceid','simid','**PhoneNumber','Crew','** >>>>> PhoneUserLab','PhoneUserOps','**PhoneUserOth','PhoneUser','**Phonename') >>>>> VALUES ('uuid:ef0b5fd9-2ebd-487c-**b451-dfeda5dd13a6','**2013060402','','06/26/2013 >>>>> 16:04:44.807','true','06/26/**2013 16:04:44.807','ver_2013-Jun-**04_rev02','06/26/2013 >>>>> 16:04:09.785','**356854040053167','**8912230000040758245','','** >>>>> PestLab','BillSmith','','','**BillSmith','Myphone bill smith') ; >>>>> INSERT INTO 1IQS48A9Tq-**R8s6geNV3yQvH4MPbaJCqsrX1861I >>>>> ('*meta-instance-id*','*meta-**model-version*','*meta-ui-** >>>>> version*','*meta-submission-**date*','*meta-is-complete*','*** >>>>> meta-date-marked-as-complete*'**,'FormVersion','DateAuto','** >>>>> deviceid','simid','**PhoneNumber','Crew','** >>>>> PhoneUserLab','PhoneUserOps','**PhoneUserOth','PhoneUser','**Phonename') >>>>> VALUES ('uuid:1db681c8-2716-4d7c-**829a-45195b45955c','**2013060402','','08/01/2013 >>>>> 18:41:53.837','true','08/01/**2013 18:41:53.837','ver_2013-Jun-**04_rev02','08/01/2013 >>>>> 18:41:34.003','**356854040053100','**8912230000127052512','** >>>>> 17809185555','PestLab','**JohnSmith','','','JohnSmith','**Mine') >>>>> >>>>> >>>>> >>>>> The last error message in the list is the shortest example of an >>>>> error that happens for more than one form. I compared the data in these >>>>> messages with existing data in the corresponding Fusion Tables and it's >>>>> always for records that I had to directly alter in the Fusion Tables. >>>>> >>>>> Purging the waiting tasks in the frontend-queue stops activity in >>>>> the Dashboard charts briefly and pausing the frontend-queue stops most >>>>> activity for as long as it is paused while still queuing tasks. 256 tasks >>>>> piled up last night. I was running version 1.3.1 and then upgraded to >>>>> 1.3.2 in hopes that might solve the problem but nothing changed. >>>>> >>>>> To me it seems like something is "stuck" and keeps resubmitting >>>>> but I really don't know how to fix it. I'm a little cautious of pushing >>>>> too many buttons without knowing what things like "Flushing Memcache" etc. >>>>> actually do. I've searched through ODK Community and Google App Engine >>>>> groups but am not finding anything similar. >>>>> >>>>> Any suggestions of what to do, information needed to solve the >>>>> problem, places to refer my question, wild-ass guesses would be greatly >>>>> appreciated. As a side question, does anyone know of good documentation on >>>>> App Engine from an administrators point of view? All the documentation >>>>> I've seen seems oriented to programming. >>>>> >>>>> Thank you. >>>>> >>>>> >>>>> >>>>> -- >>>>> -- >>>>> Post: opend...@googlegroups.com >>>>> Unsubscribe: opendatakit...@**googlegroups.com >>>>> >>>>> Options: http://groups.google.com/**group/opendatakit?hl=en >>>>> >>>>> --- >>>>> You received this message because you are subscribed to the Google >>>>> Groups "ODK Community" group. >>>>> To unsubscribe from this group and stop receiving emails from it, send >>>>> an email to opendatakit...@**googlegroups.com. >>>>> >>>>> For more options, visit https://groups.google.com/**groups/opt_out >>>>> . >>>>> >>>>> >>>>> >>>> >>>> >>>> >>>> -- >>>> Mitch Sundt >>>> Software Engineer >>>> University of Washington >>>> mitche...@gmail.com >>>> >>> >>> >>> >>> -- >>> Mitch Sundt >>> Software Engineer >>> University of Washington >>> mitche...@gmail.com >>> >> -- >> -- >> Post: opend...@googlegroups.com >> Unsubscribe: opendatakit...@googlegroups.com >> Options: http://groups.google.com/group/opendatakit?hl=en >> >> --- >> You received this message because you are subscribed to the Google Groups >> "ODK Community" group. >> To unsubscribe from this group and stop receiving emails from it, send an >> email to opendatakit...@googlegroups.com . >> For more options, visit https://groups.google.com/groups/opt_out. >> >> >> > > > > -- > Mitch Sundt > Software Engineer > University of Washington > mitche...@gmail.com >

Greg · August 7, 2013, 3:48pm

Just as a followup. I deleted the paused publishers in ODK and
re-enabled the Faster Background Actions. After about 24 hours GAE seems
to be acting normally again and not racking up charges. FYI I'm guessing
this started when Google had a "problem" around June 10 & 11 and then I had
to republish everything.

 Anyway, thank you for your help Mitch!  I greatly appreciate it.

Greg

···

On Friday, August 2, 2013 3:30:53 PM UTC-6, Greg wrote: > > Hi, > No I didn't see that, it's a shared account so thank you very much for > letting me know. I'm just leaving for the weekend but I suspect you > probably found the issue. I didn't know ODK would retain publishing links > like that. I'm assuming it does that after a form upgrade? Or would that > be after deleting a form and uploading a new one of the same name and ID. > In any case I'll have to check when I changed the form and delete the > PAUSED ones. > > Thank you very much again! > > > On Friday, August 2, 2013 1:34:08 PM UTC-6, Mitch Sundt wrote: >> >> Not sure if you saw this e-mail I sent to the edmptestlab@ gmail account: >> ---start-------------------- >> Hi, >> >> A number of your publishers are in the 'PAUSED' state which indicates >> either a transient or a ongoing failure to publish to the destination >> servers. >> >> In one instance, for this publisher: >> >> Form: Check-In Form >> To: Fusion Tables >> Establishment Date: 2013-06-05 21:26:24.969000 >> Mode: Both Uploading and Streaming >> >> The error is that the fusion table no longer exists. >> >> You should double-check that the links to the fusion tables and >> spreadsheets that are in the 'PAUSED' state actually work. If they don't, >> you should delete those publishers. >> >> Otherwise, ODK Aggregate will keep attempting to publish to those >> locations, fail, and re-attempt it a short time later. Right now, that is >> every 90 seconds. In a future release we will retry once 90 seconds later, >> then retry only every 7.5 minutes, or every 15 minutes if the server is not >> in use. >> >> --end-------------------- >> >> w.r.t. keeping the box checked -- this simply sets the behavior back to >> the highly conservative pre-1.3.1 publishing rate (a much slower rate, >> aimed at conserving quota for interactive use, and trickling data into >> Fusion Tables, etc.). Once the 7.5 minute back-off is available >> (mentioned in the above e-mail), you should not need to check this >> checkbox, as the server will then automatically shift failing publishers >> into a highly-conservative publishing-retry cycle, and move them back into >> the aggressive publishing cycle once they stop failing. >> >> Mitch >> >> On Fri, Aug 2, 2013 at 10:06 AM, Greg wrote: >> >>> Hi Mitch, >>> >>> I checked the *'D**isable faster background actions (exports, >>> publishing, form deletion) (slows quota usage on Google AppEngine)'*and that has helped immensely. The logs are still full of the ": >>> Time left on lock:" entries but the Errors have mostly disappeared and we >>> aren't getting a constant (and expensive) /gae/uploadSubmissionsTask load. >>> >>> I wasn't even aware of the validation rules in Fusion and >>> Spreadsheets. That function wasn't available from the "Change Variables" >>> section of Fusion Tables when I last set these up in June. I checked >>> through anyway and no validation rules are in place. However I do have >>> calculated columns in place. Like I said though, from the few records I >>> was able to glean from errors, the ones it keeps trying to insert and >>> getting errors on, are the same ones I've made manual changes to in Fusion >>> Tables. The spreadsheets also return errors but nothing to indicate which >>> entries are resulting in errors. Admittedly though I'm not the best at >>> sorting through the logs and mostly I just keep looking through until my >>> eyeballs bleed. I don't seem to understand the flavour of regex used. >>> >>> So as for deleting offending records, perhaps I can just delete all >>> the records of the offending tables since we use Fusion Tables for our >>> data. As for the "Disable faster background actions", what is the long >>> term consequence of keeping that box checked? I feel a little nervous >>> about keeping a stop gap measure in place without curing the initial source >>> of the problem. I'll have to contact the Fusion Tables team too but I >>> wonder if they will be like the Google App Engine team that basically said >>> Google's end is good, it's really my problem. >>> >>> I've added you as a developer to App Engine in case you want to >>> snoop around and see what's wrong or research for ODK development. Thanks >>> again for the help! >>> >>> >>> >>> >>> >>> On Thursday, August 1, 2013 4:16:44 PM UTC-6, Mitch Sundt wrote: >>> >>>> Argh! >>>> >>>> My untested suspicion is that you may have set up a validation rule or >>>> constraint, and, when Aggregate tries to submit a record that violates that >>>> constraint, Fusion Tables rejects the publishing of that submission, >>>> causing the publisher to stall and never get past the failing record (it >>>> sits in an infinite retry). >>>> >>>> If that is what is happening, the two choices are: >>>> >>>> (1) Log onto ODK Aggregate as a user with Forms Management privileges. >>>> (2) find the offending records on the submissions page (e.g., via >>>> filtering) >>>> (3) click the red 'X' to delete those records (permanently) from ODK >>>> Aggregate >>>> >>>> Or, >>>> >>>> Eliminate the validation constraints in the Fusion Table or Spreadsheet >>>> that you are publishing to. >>>> >>>> Mitch >>>> >>>> >>>> >>>> On Thu, Aug 1, 2013 at 3:13 PM, Mitch Sundt wrote: >>>> >>>>> As a stop-gap, >>>>> >>>>> log onto ODK Aggregate >>>>> Go to Site Admin / Preferences >>>>> Check the checkbox labelled *'Disable faster background actions >>>>> (exports, publishing, form deletion) (slows quota usage on Google >>>>> AppEngine)' >>>>> >>>>> * >>>>> This should restore your usage to prior values (it disables faster >>>>> publishing, restoring behavior to the pre-1.3.1 behavior). >>>>> >>>>> Do you have any validation rules set up on your fusion tables or >>>>> spreadsheets? >>>>> In the last example, 'Backend error' would indicate that it is an >>>>> error on the Fusion Tables side. Have you contacted the Fusion Tables team? >>>>> >>>>> Do you have any validation rules set up on your tables or spreadsheets? >>>>> >>>>> * >>>>> * >>>>> >>>>> >>>>> On Thu, Aug 1, 2013 at 12:35 PM, Greg wrote: >>>>> >>>>>> Hi, >>>>>> I'm running ODK Aggregate v.1.3.2 on Google App Engine and the >>>>>> frontend-queue is running 24/7 with tasks and racking up some big charges. >>>>>> Most of this time we are not using Aggregate. Google Enterprise Support >>>>>> confirmed that most of the load came from /gae/uploadSubmissionsTask & >>>>>> /gae/watchdog but claims everything is running normally on their end. >>>>>> >>>>>> >>>>>> I'm dug through the server logs and most of the Info type of >>>>>> entries are the >>>>>> >>>>>> I 2013-08-01 13:02:30.447 [s~edmpestlab/1.**369201346591658058].: >>>>>> Time left on lock: -12380028. >>>>>> >>>>>> >>>>>> These Warning messages I hadn't seen before today: >>>>>> >>>>>> W 2013-08-01 13:02:50.717 org.opendatakit.aggregate.**task.**UploadSubmissionsWorkerImpl >>>>>> uploadAllSubmissions: Upload invoked when operational status is not ACTIVE >>>>>> >>>>>> W 2013-08-01 13:02:50.953 org.opendatakit.aggregate.**task.**UploadSubmissionsWorkerImpl >>>>>> uploadAllSubmissions: Unable to acquire lock >>>>>> >>>>>> >>>>>> Examples of the error messages: >>>>>> >>>>>> E 2013-08-01 13:03:08.889 org.opendatakit.aggregate.** >>>>>> externalservice.**GoogleSpreadsheet insertData: Unable to insert >>>>>> data into spreadsheet exception: OK >>>>>> >>>>>> E 2013-08-01 13:03:09.236 org.opendatakit.aggregate.** >>>>>> task.gae.servlet.**UploadSubmissionsTaskServlet doGet: >>>>>> org.opendatakit.aggregate.**exception.**ODKExternalServiceException: >>>>>> com.go >>>>>> >>>>>> E 2013-08-01 13:01:54.411 org.opendatakit.aggregate.** >>>>>> task.gae.servlet.**UploadSubmissionsTaskServlet doGet: >>>>>> org.opendatakit.aggregate.**exception.**ODKExternalServiceException: >>>>>> { "err >>>>>> >>>>>> E 2013-08-01 13:03:07.456 >>>>>> org.opendatakit.aggregate.**task.gae.servlet.**UploadSubmissionsTaskServlet >>>>>> doGet: org.opendatakit.aggregate.**exception.**ODKExternalServiceException: >>>>>> { "error": { "errors": [ { "domain": "global", "reason": >>>>>> "backendError", "message": "Backend Error" } ], "code": 503, >>>>>> "message": "Backend Error" }}INSERT INTO 1IQS48A9Tq-**R8s6geNV3yQvH4MPbaJCqsrX1861I >>>>>> ('*meta-instance-id*','*meta-**model-version*','*meta-ui-** >>>>>> version*','*meta-submission-**date*','*meta-is-complete*','*** >>>>>> meta-date-marked-as-complete*'**,'FormVersion','DateAuto','** >>>>>> deviceid','simid','**PhoneNumber','Crew','** >>>>>> PhoneUserLab','PhoneUserOps','**PhoneUserOth','PhoneUser','**Phonename') >>>>>> VALUES ('uuid:ef0b5fd9-2ebd-487c-**b451-dfeda5dd13a6','**2013060402','','06/26/2013 >>>>>> 16:04:44.807','true','06/26/**2013 16:04:44.807','ver_2013-Jun-**04_rev02','06/26/2013 >>>>>> 16:04:09.785','**356854040053167','**8912230000040758245','','** >>>>>> PestLab','BillSmith','','','**BillSmith','Myphone bill smith') ; >>>>>> INSERT INTO 1IQS48A9Tq-**R8s6geNV3yQvH4MPbaJCqsrX1861I >>>>>> ('*meta-instance-id*','*meta-**model-version*','*meta-ui-** >>>>>> version*','*meta-submission-**date*','*meta-is-complete*','*** >>>>>> meta-date-marked-as-complete*'**,'FormVersion','DateAuto','** >>>>>> deviceid','simid','**PhoneNumber','Crew','** >>>>>> PhoneUserLab','PhoneUserOps','**PhoneUserOth','PhoneUser','**Phonename') >>>>>> VALUES ('uuid:1db681c8-2716-4d7c-**829a-45195b45955c','**2013060402','','08/01/2013 >>>>>> 18:41:53.837','true','08/01/**2013 18:41:53.837','ver_2013-Jun-**04_rev02','08/01/2013 >>>>>> 18:41:34.003','**356854040053100','**8912230000127052512','** >>>>>> 17809185555','PestLab','**JohnSmith','','','JohnSmith','**Mine') >>>>>> >>>>>> >>>>>> >>>>>> The last error message in the list is the shortest example of an >>>>>> error that happens for more than one form. I compared the data in these >>>>>> messages with existing data in the corresponding Fusion Tables and it's >>>>>> always for records that I had to directly alter in the Fusion Tables. >>>>>> >>>>>> Purging the waiting tasks in the frontend-queue stops activity >>>>>> in the Dashboard charts briefly and pausing the frontend-queue stops most >>>>>> activity for as long as it is paused while still queuing tasks. 256 tasks >>>>>> piled up last night. I was running version 1.3.1 and then upgraded to >>>>>> 1.3.2 in hopes that might solve the problem but nothing changed. >>>>>> >>>>>> To me it seems like something is "stuck" and keeps resubmitting >>>>>> but I really don't know how to fix it. I'm a little cautious of pushing >>>>>> too many buttons without knowing what things like "Flushing Memcache" etc. >>>>>> actually do. I've searched through ODK Community and Google App Engine >>>>>> groups but am not finding anything similar. >>>>>> >>>>>> Any suggestions of what to do, information needed to solve the >>>>>> problem, places to refer my question, wild-ass guesses would be greatly >>>>>> appreciated. As a side question, does anyone know of good documentation on >>>>>> App Engine from an administrators point of view? All the documentation >>>>>> I've seen seems oriented to programming. >>>>>> >>>>>> Thank you. >>>>>> >>>>>> >>>>>> >>>>>> -- >>>>>> -- >>>>>> Post: opend...@googlegroups.com >>>>>> Unsubscribe: opendatakit...@**googlegroups.com >>>>>> >>>>>> Options: http://groups.google.com/**group/opendatakit?hl=en >>>>>> >>>>>> --- >>>>>> You received this message because you are subscribed to the Google >>>>>> Groups "ODK Community" group. >>>>>> To unsubscribe from this group and stop receiving emails from it, >>>>>> send an email to opendatakit...@**googlegroups.com. >>>>>> >>>>>> For more options, visit https://groups.google.com/**groups/opt_out >>>>>> . >>>>>> >>>>>> >>>>>> >>>>> >>>>> >>>>> >>>>> -- >>>>> Mitch Sundt >>>>> Software Engineer >>>>> University of Washington >>>>> mitche...@gmail.com >>>>> >>>> >>>> >>>> >>>> -- >>>> Mitch Sundt >>>> Software Engineer >>>> University of Washington >>>> mitche...@gmail.com >>>> >>> -- >>> -- >>> Post: opend...@googlegroups.com >>> Unsubscribe: opendatakit...@googlegroups.com >>> Options: http://groups.google.com/group/opendatakit?hl=en >>> >>> --- >>> You received this message because you are subscribed to the Google >>> Groups "ODK Community" group. >>> To unsubscribe from this group and stop receiving emails from it, send >>> an email to opendatakit...@googlegroups.com. >>> For more options, visit https://groups.google.com/groups/opt_out. >>> >>> >>> >> >> >> >> -- >> Mitch Sundt >> Software Engineer >> University of Washington >> mitche...@gmail.com >> >

Mitch_S · August 7, 2013, 4:49pm

Great. Glad it resolved the issue.

Since you are using AppEngine, you should upgrade to the ODK Aggregate
1.3.3 when you have a chance.

Google has expanded the length of their unique keys and this may cause
problems with publishers should Google start giving the longer unique keys
to your submissions. If you again start having publishers in the paused
state, this will be the first thing to try.

Mitch

···

On Wed, Aug 7, 2013 at 8:48 AM, Greg wrote:

 Just as a followup.  I deleted the paused publishers in ODK and
re-enabled the Faster Background Actions. After about 24 hours GAE seems
to be acting normally again and not racking up charges. FYI I'm guessing
this started when Google had a "problem" around June 10 & 11 and then I had
to republish everything.
 Anyway, thank you for your help Mitch!  I greatly appreciate it.
Greg

On Friday, August 2, 2013 3:30:53 PM UTC-6, Greg wrote:
Hi,
No I didn't see that, it's a shared account so thank you very much
for letting me know. I'm just leaving for the weekend but I suspect you
probably found the issue. I didn't know ODK would retain publishing links
like that. I'm assuming it does that after a form upgrade? Or would that
be after deleting a form and uploading a new one of the same name and ID.
In any case I'll have to check when I changed the form and delete the
PAUSED ones.

Thank you very much again!

On Friday, August 2, 2013 1:34:08 PM UTC-6, Mitch Sundt wrote:
Not sure if you saw this e-mail I sent to the edmptestlab@ gmail
account:
---start--------------------
Hi,

A number of your publishers are in the 'PAUSED' state which indicates
either a transient or a ongoing failure to publish to the destination
servers.

In one instance, for this publisher:

Form: Check-In Form
To: Fusion Tables
Establishment Date: 2013-06-05 21:26:24.969000
Mode: Both Uploading and Streaming

The error is that the fusion table no longer exists.

You should double-check that the links to the fusion tables and
spreadsheets that are in the 'PAUSED' state actually work. If they don't,
you should delete those publishers.

Otherwise, ODK Aggregate will keep attempting to publish to those
locations, fail, and re-attempt it a short time later. Right now, that is
every 90 seconds. In a future release we will retry once 90 seconds later,
then retry only every 7.5 minutes, or every 15 minutes if the server is not
in use.

--end--------------------

w.r.t. keeping the box checked -- this simply sets the behavior back to
the highly conservative pre-1.3.1 publishing rate (a much slower rate,
aimed at conserving quota for interactive use, and trickling data into
Fusion Tables, etc.). Once the 7.5 minute back-off is available
(mentioned in the above e-mail), you should not need to check this
checkbox, as the server will then automatically shift failing publishers
into a highly-conservative publishing-retry cycle, and move them back into
the aggressive publishing cycle once they stop failing.

Mitch

On Fri, Aug 2, 2013 at 10:06 AM, Greg greg....@edmonton.ca wrote:
Hi Mitch,
 I checked the *'D**isable faster background actions (exports,
publishing, form deletion) (slows quota usage on Google AppEngine)'*and that has helped immensely. The logs are still full of the ":
Time left on lock:" entries but the Errors have mostly disappeared and we
aren't getting a constant (and expensive) /gae/uploadSubmissionsTask load.
 I wasn't even aware of the validation rules in Fusion and
Spreadsheets. That function wasn't available from the "Change Variables"
section of Fusion Tables when I last set these up in June. I checked
through anyway and no validation rules are in place. However I do have
calculated columns in place. Like I said though, from the few records I
was able to glean from errors, the ones it keeps trying to insert and
getting errors on, are the same ones I've made manual changes to in Fusion
Tables. The spreadsheets also return errors but nothing to indicate which
entries are resulting in errors. Admittedly though I'm not the best at
sorting through the logs and mostly I just keep looking through until my
eyeballs bleed. I don't seem to understand the flavour of regex used.
So as for deleting offending records, perhaps I can just delete all
the records of the offending tables since we use Fusion Tables for our
data. As for the "Disable faster background actions", what is the long
term consequence of keeping that box checked? I feel a little nervous
about keeping a stop gap measure in place without curing the initial source
of the problem. I'll have to contact the Fusion Tables team too but I
wonder if they will be like the Google App Engine team that basically said
Google's end is good, it's really my problem.
 I've added you as a developer to App Engine in case you want to
snoop around and see what's wrong or research for ODK development. Thanks
again for the help!

On Thursday, August 1, 2013 4:16:44 PM UTC-6, Mitch Sundt wrote:
Argh!

My untested suspicion is that you may have set up a validation rule or
constraint, and, when Aggregate tries to submit a record that violates that
constraint, Fusion Tables rejects the publishing of that submission,
causing the publisher to stall and never get past the failing record (it
sits in an infinite retry).

If that is what is happening, the two choices are:

(1) Log onto ODK Aggregate as a user with Forms Management privileges.
(2) find the offending records on the submissions page (e.g., via
filtering)
(3) click the red 'X' to delete those records (permanently) from ODK
Aggregate

Or,

Eliminate the validation constraints in the Fusion Table or
Spreadsheet that you are publishing to.

Mitch

On Thu, Aug 1, 2013 at 3:13 PM, Mitch Sundt mitche...@gmail.comwrote:
As a stop-gap,

log onto ODK Aggregate
Go to Site Admin / Preferences
Check the checkbox labelled *'Disable faster background actions
(exports, publishing, form deletion) (slows quota usage on Google
AppEngine)'

This should restore your usage to prior values (it disables faster
publishing, restoring behavior to the pre-1.3.1 behavior).

Do you have any validation rules set up on your fusion tables or
spreadsheets?
In the last example, 'Backend error' would indicate that it is an
error on the Fusion Tables side. Have you contacted the Fusion Tables team?

Do you have any validation rules set up on your tables or
spreadsheets?

On Thu, Aug 1, 2013 at 12:35 PM, Greg greg....@edmonton.ca wrote:
Hi,
I'm running ODK Aggregate v.1.3.2 on Google App Engine and the
frontend-queue is running 24/7 with tasks and racking up some big charges.
Most of this time we are not using Aggregate. Google Enterprise Support
confirmed that most of the load came from /gae/uploadSubmissionsTask &
/gae/watchdog but claims everything is running normally on their end.
I'm dug through the server logs and most of the Info type of
entries are the

I 2013-08-01 13:02:30.447 [s~edmpestlab/1.**36920134659165**8058].:
Time left on lock: -12380028.
 These Warning messages I hadn't seen before today:
W 2013-08-01 13:02:50.717 org.opendatakit.aggregate.task.**UploadSubmissionsWorkerImpl
uploadAllSubmissions: Upload invoked when operational status is not ACTIVE

W 2013-08-01 13:02:50.953 org.opendatakit.aggregate.task.**UploadSubmissionsWorkerImpl
uploadAllSubmissions: Unable to acquire lock
Examples of the error messages:
E 2013-08-01 13:03:08.889 org.opendatakit.aggregate.exte
rnalservice.**GoogleSpreadsheet insertData: Unable to insert data
into spreadsheet exception: OK

E 2013-08-01 13:03:09.236 org.opendatakit.aggregate.task
.gae.servlet.UploadSubmissionsTaskServlet doGet:
org.opendatakit.aggregate.exception.ODKExternalServiceException:
com.go

E 2013-08-01 13:01:54.411 org.opendatakit.aggregate.task
.gae.servlet.UploadSubmissionsTaskServlet doGet:
org.opendatakit.aggregate.exception.ODKExternalServiceException:
{ "err

E 2013-08-01 13:03:07.456
org.opendatakit.aggregate.task.gae.servlet.UploadSubmissionsTaskServlet
doGet: org.opendatakit.aggregate.exception.**
ODKExternalServiceException: { "error": { "errors": [ {
"domain": "global", "reason": "backendError", "message": "Backend
Error" } ], "code": 503, "message": "Backend Error" }}INSERT INTO
1IQS48A9Tq-R8s6geNV3yQvH4MPbaJCqsrX1861I
('meta-instance-id','meta-model-version','meta-ui-versi
on','*meta-submission-date','meta-is-complete','meta-
date-marked-as-complete'*,'**FormVersion','DateAuto','devic
eid','simid','*PhoneNumber','Crew','PhoneUserLab','
PhoneUserOps','PhoneUserOth','PhoneUser','Phonename') VALUES
('uuid:ef0b5fd9-2ebd-487c-b451-dfeda5dd13a6','2013060402','','06/26/2013
16:04:44.807','true','06/26/2013 16:04:44.807','ver_2013-Jun-
04_rev02','06/26/2013 16:04:09.785','356854040053167','
8912230000040758245','','PestLab','BillSmith','','','BillSmith','Myphone
bill smith') ; INSERT INTO 1IQS48A9Tq-R8s6geNV3yQvH4MPbaJCqsrX1861I
('meta-instance-id','meta-model-version','meta-ui-versi
on','*meta-submission-date','meta-is-complete','meta-
date-marked-as-complete','**FormVersion','DateAuto','devic
eid','simid','**PhoneNumber','**Crew','PhoneUserLab','
PhoneUserOps','PhoneUserOth','PhoneUser','Phonename') VALUES
('uuid:1db681c8-2716-4d7c-829a-45195b45955c','2013060402','','08/01/2013
18:41:53.837','true','08/01/2013 18:41:53.837','ver_2013-Jun-
04_rev02','08/01/2013 18:41:34.003','356854040053100','
8912230000127052512','17809185555','PestLab','JohnSmith',
'','','JohnSmith','**Mine')
 The last error message in the list is the shortest example of
an error that happens for more than one form. I compared the data in these
messages with existing data in the corresponding Fusion Tables and it's
always for records that I had to directly alter in the Fusion Tables.
 Purging the waiting tasks in the frontend-queue stops activity
in the Dashboard charts briefly and pausing the frontend-queue stops most
activity for as long as it is paused while still queuing tasks. 256 tasks
piled up last night. I was running version 1.3.1 and then upgraded to
1.3.2 in hopes that might solve the problem but nothing changed.
 To me it seems like something is "stuck" and keeps resubmitting
but I really don't know how to fix it. I'm a little cautious of pushing
too many buttons without knowing what things like "Flushing Memcache" etc.
actually do. I've searched through ODK Community and Google App Engine
groups but am not finding anything similar.
 Any suggestions of what to do, information needed to solve the
problem, places to refer my question, wild-ass guesses would be greatly
appreciated. As a side question, does anyone know of good documentation on
App Engine from an administrators point of view? All the documentation
I've seen seems oriented to programming.

Thank you.

--

Post: opend...@googlegroups.com
Unsubscribe: opendatakit...@**googlegroups.**com

Options: http://groups.google.com/**group**/opendatakit?hl=en http://groups.google.com/group/opendatakit?hl=en

You received this message because you are subscribed to the Google
Groups "ODK Community" group.
To unsubscribe from this group and stop receiving emails from it,
send an email to opendatakit...@**googlegroups.**com.

For more options, visit https://groups.google.com/**grou**ps/opt_out https://groups.google.com/groups/opt_out
.
--
Mitch Sundt
Software Engineer
University of Washington
mitche...@gmail.com
--
Mitch Sundt
Software Engineer
University of Washington
mitche...@gmail.com
--

Post: opend...@googlegroups.com
Unsubscribe: opendatakit...@googlegroups.**com
Options: http://groups.google.com/**group/opendatakit?hl=en http://groups.google.com/group/opendatakit?hl=en

You received this message because you are subscribed to the Google
Groups "ODK Community" group.
To unsubscribe from this group and stop receiving emails from it, send
an email to opendatakit...@googlegroups.**com.
For more options, visit https://groups.google.com/**groups/opt_out https://groups.google.com/groups/opt_out
.
--
Mitch Sundt
Software Engineer
University of Washington
mitche...@gmail.com
--
--
Post: opendatakit@googlegroups.com
Unsubscribe: opendatakit+unsubscribe@googlegroups.com
Options: http://groups.google.com/group/opendatakit?hl=en
You received this message because you are subscribed to the Google Groups
"ODK Community" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to opendatakit+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

--
Mitch Sundt
Software Engineer
University of Washington
mitchellsundt@gmail.com

Greg · August 9, 2013, 9:20pm

Done! Thank you for the warning. Sometimes it can be difficult keeping
track of Google's updates.

···

On Wed, Aug 7, 2013 at 10:49 AM, Mitch Sundt wrote:

Great. Glad it resolved the issue.

Since you are using AppEngine, you should upgrade to the ODK Aggregate
1.3.3 when you have a chance.

Google has expanded the length of their unique keys and this may cause
problems with publishers should Google start giving the longer unique keys
to your submissions. If you again start having publishers in the paused
state, this will be the first thing to try.

Mitch

On Wed, Aug 7, 2013 at 8:48 AM, Greg greg.pommen@edmonton.ca wrote:
 Just as a followup.  I deleted the paused publishers in ODK and
re-enabled the Faster Background Actions. After about 24 hours GAE seems
to be acting normally again and not racking up charges. FYI I'm guessing
this started when Google had a "problem" around June 10 & 11 and then I had
to republish everything.
 Anyway, thank you for your help Mitch!  I greatly appreciate it.
Greg

On Friday, August 2, 2013 3:30:53 PM UTC-6, Greg wrote:
Hi,
No I didn't see that, it's a shared account so thank you very much
for letting me know. I'm just leaving for the weekend but I suspect you
probably found the issue. I didn't know ODK would retain publishing links
like that. I'm assuming it does that after a form upgrade? Or would that
be after deleting a form and uploading a new one of the same name and ID.
In any case I'll have to check when I changed the form and delete the
PAUSED ones.

Thank you very much again!

On Friday, August 2, 2013 1:34:08 PM UTC-6, Mitch Sundt wrote:
Not sure if you saw this e-mail I sent to the edmptestlab@ gmail
account:
---start--------------------
Hi,

A number of your publishers are in the 'PAUSED' state which indicates
either a transient or a ongoing failure to publish to the destination
servers.

In one instance, for this publisher:

Form: Check-In Form
To: Fusion Tables
Establishment Date: 2013-06-05 21:26:24.969000
Mode: Both Uploading and Streaming

The error is that the fusion table no longer exists.

You should double-check that the links to the fusion tables and
spreadsheets that are in the 'PAUSED' state actually work. If they don't,
you should delete those publishers.

Otherwise, ODK Aggregate will keep attempting to publish to those
locations, fail, and re-attempt it a short time later. Right now, that is
every 90 seconds. In a future release we will retry once 90 seconds later,
then retry only every 7.5 minutes, or every 15 minutes if the server is not
in use.

--end--------------------

w.r.t. keeping the box checked -- this simply sets the behavior back to
the highly conservative pre-1.3.1 publishing rate (a much slower rate,
aimed at conserving quota for interactive use, and trickling data into
Fusion Tables, etc.). Once the 7.5 minute back-off is available
(mentioned in the above e-mail), you should not need to check this
checkbox, as the server will then automatically shift failing publishers
into a highly-conservative publishing-retry cycle, and move them back into
the aggressive publishing cycle once they stop failing.

Mitch

On Fri, Aug 2, 2013 at 10:06 AM, Greg greg....@edmonton.ca wrote:
Hi Mitch,
 I checked the *'D**isable faster background actions (exports,
publishing, form deletion) (slows quota usage on Google AppEngine)'*and that has helped immensely. The logs are still full of the ":
Time left on lock:" entries but the Errors have mostly disappeared and we
aren't getting a constant (and expensive) /gae/uploadSubmissionsTask load.
 I wasn't even aware of the validation rules in Fusion and
Spreadsheets. That function wasn't available from the "Change Variables"
section of Fusion Tables when I last set these up in June. I checked
through anyway and no validation rules are in place. However I do have
calculated columns in place. Like I said though, from the few records I
was able to glean from errors, the ones it keeps trying to insert and
getting errors on, are the same ones I've made manual changes to in Fusion
Tables. The spreadsheets also return errors but nothing to indicate which
entries are resulting in errors. Admittedly though I'm not the best at
sorting through the logs and mostly I just keep looking through until my
eyeballs bleed. I don't seem to understand the flavour of regex used.
So as for deleting offending records, perhaps I can just delete
all the records of the offending tables since we use Fusion Tables for our
data. As for the "Disable faster background actions", what is the long
term consequence of keeping that box checked? I feel a little nervous
about keeping a stop gap measure in place without curing the initial source
of the problem. I'll have to contact the Fusion Tables team too but I
wonder if they will be like the Google App Engine team that basically said
Google's end is good, it's really my problem.
 I've added you as a developer to App Engine in case you want to
snoop around and see what's wrong or research for ODK development. Thanks
again for the help!

On Thursday, August 1, 2013 4:16:44 PM UTC-6, Mitch Sundt wrote:
Argh!

My untested suspicion is that you may have set up a validation rule
or constraint, and, when Aggregate tries to submit a record that violates
that constraint, Fusion Tables rejects the publishing of that submission,
causing the publisher to stall and never get past the failing record (it
sits in an infinite retry).

If that is what is happening, the two choices are:

(1) Log onto ODK Aggregate as a user with Forms Management
privileges.
(2) find the offending records on the submissions page (e.g., via
filtering)
(3) click the red 'X' to delete those records (permanently) from ODK
Aggregate

Or,

Eliminate the validation constraints in the Fusion Table or
Spreadsheet that you are publishing to.

Mitch

On Thu, Aug 1, 2013 at 3:13 PM, Mitch Sundt mitche...@gmail.comwrote:
As a stop-gap,

log onto ODK Aggregate
Go to Site Admin / Preferences
Check the checkbox labelled *'Disable faster background actions
(exports, publishing, form deletion) (slows quota usage on Google
AppEngine)'

This should restore your usage to prior values (it disables faster
publishing, restoring behavior to the pre-1.3.1 behavior).

Do you have any validation rules set up on your fusion tables or
spreadsheets?
In the last example, 'Backend error' would indicate that it is an
error on the Fusion Tables side. Have you contacted the Fusion Tables team?

Do you have any validation rules set up on your tables or
spreadsheets?

On Thu, Aug 1, 2013 at 12:35 PM, Greg greg....@edmonton.ca wrote:
Hi,
I'm running ODK Aggregate v.1.3.2 on Google App Engine and the
frontend-queue is running 24/7 with tasks and racking up some big charges.
Most of this time we are not using Aggregate. Google Enterprise Support
confirmed that most of the load came from /gae/uploadSubmissionsTask &
/gae/watchdog but claims everything is running normally on their end.
I'm dug through the server logs and most of the Info type of
entries are the

I 2013-08-01 13:02:30.447 [s~edmpestlab/1.**36920134659165**8058].:
Time left on lock: -12380028.
 These Warning messages I hadn't seen before today:
W 2013-08-01 13:02:50.717 org.opendatakit.aggregate.task.**UploadSubmissionsWorkerImpl
uploadAllSubmissions: Upload invoked when operational status is not ACTIVE

W 2013-08-01 13:02:50.953 org.opendatakit.aggregate.task.**UploadSubmissionsWorkerImpl
uploadAllSubmissions: Unable to acquire lock
Examples of the error messages:
E 2013-08-01 13:03:08.889 org.opendatakit.aggregate.exte
rnalservice.**GoogleSpreadsheet insertData: Unable to insert data
into spreadsheet exception: OK

E 2013-08-01 13:03:09.236 org.opendatakit.aggregate.task
.gae.servlet.UploadSubmissionsTaskServlet doGet:
org.opendatakit.aggregate.exception.*ODKExternalServiceExcept
*ion: com.go

E 2013-08-01 13:01:54.411 org.opendatakit.aggregate.task
.gae.servlet.UploadSubmissionsTaskServlet doGet:
org.opendatakit.aggregate.exception.*ODKExternalServiceExcept
*ion: { "err

E 2013-08-01 13:03:07.456
org.opendatakit.aggregate.task.gae.servlet.UploadSubmissions
TaskServlet doGet: org.opendatakit.aggregate.exception.
ODKExternalServiceException: { "error": { "errors": [ {
"domain": "global", "reason": "backendError", "message": "Backend
Error" } ], "code": 503, "message": "Backend Error" }}INSERT INTO
1IQS48A9Tq-R8s6geNV3yQvH4MPbaJCqsrX1861I
('meta-instance-id','meta-model-version','*meta-ui-versi
on','*meta-submission-date','meta-is-complete','meta-
date-marked-as-complete','**FormVersion','DateAuto','devic
eid','simid','**PhoneNumber','Crew','PhoneUserLab','
PhoneUserOps','PhoneUserOth','PhoneUser','Phonename')
VALUES ('uuid:ef0b5fd9-2ebd-487c-b451-dfeda5dd13a6','
2013060402','','06/26/2013 16:04:44.807','true','06/26/2013
16:04:44.807','ver_2013-Jun-04_rev02','06/26/2013
16:04:09.785','356854040053167','8912230000040758245','','
PestLab','BillSmith','','','BillSmith','Myphone bill smith')
; INSERT INTO 1IQS48A9Tq-R8s6geNV3yQvH4MPbaJCqsrX1861I
('meta-instance-id','meta-model-version','*meta-ui-versi
on','*meta-submission-date','meta-is-complete','meta-
date-marked-as-complete','**FormVersion','DateAuto','devic
eid','simid','**PhoneNumber','Crew','PhoneUserLab','
PhoneUserOps','PhoneUserOth','PhoneUser','Phonename')
VALUES ('uuid:1db681c8-2716-4d7c-829a-45195b45955c','
2013060402','','08/01/2013 18:41:53.837','true','08/01/2013
18:41:53.837','ver_2013-Jun-04_rev02','08/01/2013
18:41:34.003','356854040053100','8912230000127052512','
17809185555','PestLab','JohnSmith','','','JohnSmith','
Mine')
 The last error message in the list is the shortest example of
an error that happens for more than one form. I compared the data in these
messages with existing data in the corresponding Fusion Tables and it's
always for records that I had to directly alter in the Fusion Tables.
 Purging the waiting tasks in the frontend-queue stops activity
in the Dashboard charts briefly and pausing the frontend-queue stops most
activity for as long as it is paused while still queuing tasks. 256 tasks
piled up last night. I was running version 1.3.1 and then upgraded to
1.3.2 in hopes that might solve the problem but nothing changed.
 To me it seems like something is "stuck" and keeps
resubmitting but I really don't know how to fix it. I'm a little cautious
of pushing too many buttons without knowing what things like "Flushing
Memcache" etc. actually do. I've searched through ODK Community and Google
App Engine groups but am not finding anything similar.
 Any suggestions of what to do, information needed to solve the
problem, places to refer my question, wild-ass guesses would be greatly
appreciated. As a side question, does anyone know of good documentation on
App Engine from an administrators point of view? All the documentation
I've seen seems oriented to programming.

Thank you.

--

Post: opend...@googlegroups.com
Unsubscribe: opendatakit...@**googlegroups.**com

Options: http://groups.google.com/**group**/opendatakit?hl=en http://groups.google.com/group/opendatakit?hl=en

You received this message because you are subscribed to the Google
Groups "ODK Community" group.
To unsubscribe from this group and stop receiving emails from it,
send an email to opendatakit...@**googlegroups.**com.

For more options, visit https://groups.google.com/**grou**
ps/opt_out https://groups.google.com/groups/opt_out.
--
Mitch Sundt
Software Engineer
University of Washington
mitche...@gmail.com
--
Mitch Sundt
Software Engineer
University of Washington
mitche...@gmail.com
--

Post: opend...@googlegroups.com
Unsubscribe: opendatakit...@googlegroups.**com
Options: http://groups.google.com/**group/opendatakit?hl=en http://groups.google.com/group/opendatakit?hl=en

You received this message because you are subscribed to the Google
Groups "ODK Community" group.
To unsubscribe from this group and stop receiving emails from it, send
an email to opendatakit...@googlegroups.**com.
For more options, visit https://groups.google.com/**groups/opt_out https://groups.google.com/groups/opt_out
.
--
Mitch Sundt
Software Engineer
University of Washington
mitche...@gmail.com
--
--
Post: opendatakit@googlegroups.com
Unsubscribe: opendatakit+unsubscribe@googlegroups.com
Options: http://groups.google.com/group/opendatakit?hl=en
You received this message because you are subscribed to the Google Groups
"ODK Community" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to opendatakit+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.
--
Mitch Sundt
Software Engineer
University of Washington
mitchellsundt@gmail.com

--

Post: opendatakit@googlegroups.com
Unsubscribe: opendatakit+unsubscribe@googlegroups.com
Options: http://groups.google.com/group/opendatakit?hl=en

You received this message because you are subscribed to a topic in the
Google Groups "ODK Community" group.
To unsubscribe from this topic, visit
https://groups.google.com/d/topic/opendatakit/MBzQgOHKVNc/unsubscribe.
To unsubscribe from this group and all its topics, send an email to
opendatakit+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

--
Greg Pommen
Biological Sciences Technical Assistant
City of Edmonton Environmental Services
12304-107 St.
Edmonton, AB T5G 2S7
(780) 496-6906
fax: (780) 496-4978
greg.pommen@edmonton.ca

John_Berini · February 6, 2015, 6:48pm

Hi There,

I'm running into the same issue that Greg ran into last year. In mid
January, I tried exporting all my uploaded data from ODK Aggregate to a
google fusion table, and I've been having major issues ever since. My
Datastore Read Operations quota on Google's AppEngine has been maxing out
everyday for roughly the last month, and as a result I haven't been able to
login to ODK Aggregate. In the last 24 hours AppEngine has logged 317
errors due to "Upload Submissions Tasks", and this has been occurring daily
since the time I tried publishing my data to a Google Fusion Table (January
14th). I tried completely disabling the app on Jan 22, and I've also
completely deleted the Fusion Table in an attempt to discontinue any
looping tasks that might be associated with the table.

Nothing that I've tried appears to be working. Any assistance anyone can
offer would be greatly appreciated.

John

···

On Friday, August 9, 2013 at 4:20:24 PM UTC-5, Greg wrote: > > Done! Thank you for the warning. Sometimes it can be difficult keeping > track of Google's updates. > > > On Wed, Aug 7, 2013 at 10:49 AM, Mitch Sundt <mitche...@gmail.com > wrote: > >> Great. Glad it resolved the issue. >> >> Since you are using AppEngine, you should upgrade to the ODK Aggregate >> 1.3.3 when you have a chance. >> >> Google has expanded the length of their unique keys and this may cause >> problems with publishers should Google start giving the longer unique keys >> to your submissions. If you again start having publishers in the paused >> state, this will be the first thing to try. >> >> Mitch >> >> >> >> On Wed, Aug 7, 2013 at 8:48 AM, Greg <greg....@edmonton.ca > wrote: >> >>> Just as a followup. I deleted the paused publishers in ODK and >>> re-enabled the Faster Background Actions. After about 24 hours GAE seems >>> to be acting normally again and not racking up charges. FYI I'm guessing >>> this started when Google had a "problem" around June 10 & 11 and then I had >>> to republish everything. >>> >>> Anyway, thank you for your help Mitch! I greatly appreciate it. >>> >>> Greg >>> >>> >>> On Friday, August 2, 2013 3:30:53 PM UTC-6, Greg wrote: >>>> >>>> Hi, >>>> No I didn't see that, it's a shared account so thank you very much >>>> for letting me know. I'm just leaving for the weekend but I suspect you >>>> probably found the issue. I didn't know ODK would retain publishing links >>>> like that. I'm assuming it does that after a form upgrade? Or would that >>>> be after deleting a form and uploading a new one of the same name and ID. >>>> In any case I'll have to check when I changed the form and delete the >>>> PAUSED ones. >>>> >>>> Thank you very much again! >>>> >>>> >>>> On Friday, August 2, 2013 1:34:08 PM UTC-6, Mitch Sundt wrote: >>>>> >>>>> Not sure if you saw this e-mail I sent to the edmptestlab@ gmail >>>>> account: >>>>> ---start-------------------- >>>>> Hi, >>>>> >>>>> A number of your publishers are in the 'PAUSED' state which indicates >>>>> either a transient or a ongoing failure to publish to the destination >>>>> servers. >>>>> >>>>> In one instance, for this publisher: >>>>> >>>>> Form: Check-In Form >>>>> To: Fusion Tables >>>>> Establishment Date: 2013-06-05 21:26:24.969000 >>>>> Mode: Both Uploading and Streaming >>>>> >>>>> The error is that the fusion table no longer exists. >>>>> >>>>> You should double-check that the links to the fusion tables and >>>>> spreadsheets that are in the 'PAUSED' state actually work. If they don't, >>>>> you should delete those publishers. >>>>> >>>>> Otherwise, ODK Aggregate will keep attempting to publish to those >>>>> locations, fail, and re-attempt it a short time later. Right now, that is >>>>> every 90 seconds. In a future release we will retry once 90 seconds later, >>>>> then retry only every 7.5 minutes, or every 15 minutes if the server is not >>>>> in use. >>>>> >>>>> --end-------------------- >>>>> >>>>> w.r.t. keeping the box checked -- this simply sets the behavior back >>>>> to the highly conservative pre-1.3.1 publishing rate (a much slower rate, >>>>> aimed at conserving quota for interactive use, and trickling data into >>>>> Fusion Tables, etc.). Once the 7.5 minute back-off is available >>>>> (mentioned in the above e-mail), you should not need to check this >>>>> checkbox, as the server will then automatically shift failing publishers >>>>> into a highly-conservative publishing-retry cycle, and move them back into >>>>> the aggressive publishing cycle once they stop failing. >>>>> >>>>> Mitch >>>>> >>>>> On Fri, Aug 2, 2013 at 10:06 AM, Greg wrote: >>>>> >>>>>> Hi Mitch, >>>>>> >>>>>> I checked the *'D**isable faster background actions (exports, >>>>>> publishing, form deletion) (slows quota usage on Google AppEngine)'* >>>>>> and that has helped immensely. The logs are still full of the ": >>>>>> Time left on lock:" entries but the Errors have mostly disappeared and we >>>>>> aren't getting a constant (and expensive) /gae/uploadSubmissionsTask load. >>>>>> >>>>>> I wasn't even aware of the validation rules in Fusion and >>>>>> Spreadsheets. That function wasn't available from the "Change Variables" >>>>>> section of Fusion Tables when I last set these up in June. I checked >>>>>> through anyway and no validation rules are in place. However I do have >>>>>> calculated columns in place. Like I said though, from the few records I >>>>>> was able to glean from errors, the ones it keeps trying to insert and >>>>>> getting errors on, are the same ones I've made manual changes to in Fusion >>>>>> Tables. The spreadsheets also return errors but nothing to indicate which >>>>>> entries are resulting in errors. Admittedly though I'm not the best at >>>>>> sorting through the logs and mostly I just keep looking through until my >>>>>> eyeballs bleed. I don't seem to understand the flavour of regex used. >>>>>> >>>>>> So as for deleting offending records, perhaps I can just delete >>>>>> all the records of the offending tables since we use Fusion Tables for our >>>>>> data. As for the "Disable faster background actions", what is the long >>>>>> term consequence of keeping that box checked? I feel a little nervous >>>>>> about keeping a stop gap measure in place without curing the initial source >>>>>> of the problem. I'll have to contact the Fusion Tables team too but I >>>>>> wonder if they will be like the Google App Engine team that basically said >>>>>> Google's end is good, it's really my problem. >>>>>> >>>>>> I've added you as a developer to App Engine in case you want to >>>>>> snoop around and see what's wrong or research for ODK development. Thanks >>>>>> again for the help! >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> On Thursday, August 1, 2013 4:16:44 PM UTC-6, Mitch Sundt wrote: >>>>>> >>>>>>> Argh! >>>>>>> >>>>>>> My untested suspicion is that you may have set up a validation rule >>>>>>> or constraint, and, when Aggregate tries to submit a record that violates >>>>>>> that constraint, Fusion Tables rejects the publishing of that submission, >>>>>>> causing the publisher to stall and never get past the failing record (it >>>>>>> sits in an infinite retry). >>>>>>> >>>>>>> If that is what is happening, the two choices are: >>>>>>> >>>>>>> (1) Log onto ODK Aggregate as a user with Forms Management >>>>>>> privileges. >>>>>>> (2) find the offending records on the submissions page (e.g., via >>>>>>> filtering) >>>>>>> (3) click the red 'X' to delete those records (permanently) from ODK >>>>>>> Aggregate >>>>>>> >>>>>>> Or, >>>>>>> >>>>>>> Eliminate the validation constraints in the Fusion Table or >>>>>>> Spreadsheet that you are publishing to. >>>>>>> >>>>>>> Mitch >>>>>>> >>>>>>> >>>>>>> >>>>>>> On Thu, Aug 1, 2013 at 3:13 PM, Mitch Sundt wrote: >>>>>>> >>>>>>>> As a stop-gap, >>>>>>>> >>>>>>>> log onto ODK Aggregate >>>>>>>> Go to Site Admin / Preferences >>>>>>>> Check the checkbox labelled >>>>>>>> >>>>>>>> *'Disable faster background actions (exports, publishing, form >>>>>>>> deletion) (slows quota usage on Google AppEngine)' * >>>>>>>> This should restore your usage to prior values (it disables faster >>>>>>>> publishing, restoring behavior to the pre-1.3.1 behavior). >>>>>>>> >>>>>>>> Do you have any validation rules set up on your fusion tables or >>>>>>>> spreadsheets? >>>>>>>> In the last example, 'Backend error' would indicate that it is an >>>>>>>> error on the Fusion Tables side. Have you contacted the Fusion Tables team? >>>>>>>> >>>>>>>> Do you have any validation rules set up on your tables or >>>>>>>> spreadsheets? >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> On Thu, Aug 1, 2013 at 12:35 PM, Greg wrote: >>>>>>>> >>>>>>>>> Hi, >>>>>>>>> I'm running ODK Aggregate v.1.3.2 on Google App Engine and the >>>>>>>>> frontend-queue is running 24/7 with tasks and racking up some big charges. >>>>>>>>> Most of this time we are not using Aggregate. Google Enterprise Support >>>>>>>>> confirmed that most of the load came from /gae/uploadSubmissionsTask & >>>>>>>>> /gae/watchdog but claims everything is running normally on their end. >>>>>>>>> >>>>>>>>> >>>>>>>>> I'm dug through the server logs and most of the Info type of >>>>>>>>> entries are the >>>>>>>>> >>>>>>>>> I 2013-08-01 13:02:30.447 [s~edmpestlab/1.369201346591658058].: >>>>>>>>> Time left on lock: -12380028. >>>>>>>>> >>>>>>>>> >>>>>>>>> These Warning messages I hadn't seen before today: >>>>>>>>> >>>>>>>>> W 2013-08-01 13:02:50.717 org.opendatakit.aggregate.task.UploadSubmissionsWorkerImpl >>>>>>>>> uploadAllSubmissions: Upload invoked when operational status is not ACTIVE >>>>>>>>> >>>>>>>>> W 2013-08-01 13:02:50.953 org.opendatakit.aggregate.task.UploadSubmissionsWorkerImpl >>>>>>>>> uploadAllSubmissions: Unable to acquire lock >>>>>>>>> >>>>>>>>> >>>>>>>>> Examples of the error messages: >>>>>>>>> >>>>>>>>> E 2013-08-01 13:03:08.889 org.opendatakit.aggregate.exte >>>>>>>>> rnalservice.GoogleSpreadsheet insertData: Unable to insert data >>>>>>>>> into spreadsheet exception: OK >>>>>>>>> >>>>>>>>> E 2013-08-01 13:03:09.236 org.opendatakit.aggregate.task >>>>>>>>> .gae.servlet.UploadSubmissionsTaskServlet doGet: >>>>>>>>> org.opendatakit.aggregate.exception.ODKExternalServiceException: >>>>>>>>> com.go >>>>>>>>> >>>>>>>>> E 2013-08-01 13:01:54.411 org.opendatakit.aggregate.task >>>>>>>>> .gae.servlet.UploadSubmissionsTaskServlet doGet: >>>>>>>>> org.opendatakit.aggregate.exception.ODKExternalServiceException: >>>>>>>>> { "err >>>>>>>>> >>>>>>>>> E 2013-08-01 13:03:07.456 >>>>>>>>> org.opendatakit.aggregate.task.gae.servlet.UploadSubmissionsTaskServlet >>>>>>>>> doGet: org.opendatakit.aggregate.exception. >>>>>>>>> ODKExternalServiceException: { "error": { "errors": [ { >>>>>>>>> "domain": "global", "reason": "backendError", "message": "Backend >>>>>>>>> Error" } ], "code": 503, "message": "Backend Error" }}INSERT INTO >>>>>>>>> 1IQS48A9Tq-R8s6geNV3yQvH4MPbaJCqsrX1861I >>>>>>>>> ('*meta-instance-id*','*meta-model-version*','*meta-ui-versi >>>>>>>>> on*','*meta-submission-date*','*meta-is-complete*','*meta- >>>>>>>>> date-marked-as-complete*','FormVersion','DateAuto','devic >>>>>>>>> eid','simid','PhoneNumber','Crew','PhoneUserLab','PhoneUserOps',' >>>>>>>>> PhoneUserOth','PhoneUser','Phonename') VALUES >>>>>>>>> ('uuid:ef0b5fd9-2ebd-487c-b451-dfeda5dd13a6','2013060402','','06/26/2013 >>>>>>>>> 16:04:44.807','true','06/26/2013 16:04:44.807','ver_2013-Jun-04_rev02','06/26/2013 >>>>>>>>> 16:04:09.785','356854040053167','8912230000040758245','','Pe >>>>>>>>> stLab','BillSmith','','','BillSmith','Myphone bill smith') ; >>>>>>>>> INSERT INTO 1IQS48A9Tq-R8s6geNV3yQvH4MPbaJCqsrX1861I >>>>>>>>> ('*meta-instance-id*','*meta-model-version*','*meta-ui-versi >>>>>>>>> on*','*meta-submission-date*','*meta-is-complete*','*meta- >>>>>>>>> date-marked-as-complete*','FormVersion','DateAuto','devic >>>>>>>>> eid','simid','PhoneNumber','Crew','PhoneUserLab','PhoneUserOps',' >>>>>>>>> PhoneUserOth','PhoneUser','Phonename') VALUES >>>>>>>>> ('uuid:1db681c8-2716-4d7c-829a-45195b45955c','2013060402','','08/01/2013 >>>>>>>>> 18:41:53.837','true','08/01/2013 18:41:53.837','ver_2013-Jun-04_rev02','08/01/2013 >>>>>>>>> 18:41:34.003','356854040053100','8912230000127052512','17809 >>>>>>>>> 185555','PestLab','JohnSmith','','','JohnSmith','Mine') >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> The last error message in the list is the shortest example of >>>>>>>>> an error that happens for more than one form. I compared the data in these >>>>>>>>> messages with existing data in the corresponding Fusion Tables and it's >>>>>>>>> always for records that I had to directly alter in the Fusion Tables. >>>>>>>>> >>>>>>>>> Purging the waiting tasks in the frontend-queue stops >>>>>>>>> activity in the Dashboard charts briefly and pausing the frontend-queue >>>>>>>>> stops most activity for as long as it is paused while still queuing tasks. >>>>>>>>> 256 tasks piled up last night. I was running version 1.3.1 and then >>>>>>>>> upgraded to 1.3.2 in hopes that might solve the problem but nothing changed. >>>>>>>>> >>>>>>>>> To me it seems like something is "stuck" and keeps >>>>>>>>> resubmitting but I really don't know how to fix it. I'm a little cautious >>>>>>>>> of pushing too many buttons without knowing what things like "Flushing >>>>>>>>> Memcache" etc. actually do. I've searched through ODK Community and Google >>>>>>>>> App Engine groups but am not finding anything similar. >>>>>>>>> >>>>>>>>> Any suggestions of what to do, information needed to solve >>>>>>>>> the problem, places to refer my question, wild-ass guesses would be greatly >>>>>>>>> appreciated. As a side question, does anyone know of good documentation on >>>>>>>>> App Engine from an administrators point of view? All the documentation >>>>>>>>> I've seen seems oriented to programming. >>>>>>>>> >>>>>>>>> Thank you. >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> -- >>>>>>>>> -- >>>>>>>>> Post: opend...@googlegroups.com >>>>>>>>> Unsubscribe: opendatakit...@googlegroups.com >>>>>>>>> >>>>>>>>> Options: http://groups.google.com/group/opendatakit?hl=en >>>>>>>>> >>>>>>>>> --- >>>>>>>>> You received this message because you are subscribed to the Google >>>>>>>>> Groups "ODK Community" group. >>>>>>>>> To unsubscribe from this group and stop receiving emails from it, >>>>>>>>> send an email to opendatakit...@googlegroups.com. >>>>>>>>> >>>>>>>>> For more options, visit https://groups.google.com/groups/opt_out. >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> -- >>>>>>>> Mitch Sundt >>>>>>>> Software Engineer >>>>>>>> University of Washington >>>>>>>> mitche...@gmail.com >>>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> -- >>>>>>> Mitch Sundt >>>>>>> Software Engineer >>>>>>> University of Washington >>>>>>> mitche...@gmail.com >>>>>>> >>>>>> -- >>>>>> -- >>>>>> Post: opend...@googlegroups.com >>>>>> Unsubscribe: opendatakit...@googlegroups.com >>>>>> Options: http://groups.google.com/group/opendatakit?hl=en >>>>>> >>>>>> --- >>>>>> You received this message because you are subscribed to the Google >>>>>> Groups "ODK Community" group. >>>>>> To unsubscribe from this group and stop receiving emails from it, >>>>>> send an email to opendatakit...@googlegroups.com. >>>>>> For more options, visit https://groups.google.com/groups/opt_out. >>>>>> >>>>>> >>>>>> >>>>> >>>>> >>>>> >>>>> -- >>>>> Mitch Sundt >>>>> Software Engineer >>>>> University of Washington >>>>> mitche...@gmail.com >>>>> >>>> -- >>> -- >>> Post: opend...@googlegroups.com >>> Unsubscribe: opendatakit...@googlegroups.com >>> Options: http://groups.google.com/group/opendatakit?hl=en >>> >>> --- >>> You received this message because you are subscribed to the Google >>> Groups "ODK Community" group. >>> To unsubscribe from this group and stop receiving emails from it, send >>> an email to opendatakit...@googlegroups.com . >>> For more options, visit https://groups.google.com/groups/opt_out. >>> >>> >>> >> >> >> >> -- >> Mitch Sundt >> Software Engineer >> University of Washington >> mitche...@gmail.com >> >> -- >> -- >> Post: opend...@googlegroups.com >> Unsubscribe: opendatakit...@googlegroups.com >> Options: http://groups.google.com/group/opendatakit?hl=en >> >> --- >> You received this message because you are subscribed to a topic in the >> Google Groups "ODK Community" group. >> To unsubscribe from this topic, visit >> https://groups.google.com/d/topic/opendatakit/MBzQgOHKVNc/unsubscribe. >> To unsubscribe from this group and all its topics, send an email to >> opendatakit...@googlegroups.com . >> For more options, visit https://groups.google.com/groups/opt_out. >> >> >> > > > > -- > Greg Pommen > Biological Sciences Technical Assistant > City of Edmonton Environmental Services > 12304-107 St. > Edmonton, AB T5G 2S7 > (780) 496-6906 > fax: (780) 496-4978 > greg....@edmonton.ca >

Mitch_S · February 6, 2015, 7:03pm

Did you delete the publisher?

This can occur if you delete the fusion table, but not the publisher.

It can also happen if a submission is corrupted -- you would notice this by
the publisher never publishing new data into your fusion table.

Things to try:

(1) if you deleted the fusion table, make sure you deleted the publisher
into it.

(2) upgrade to the latest version of ODK Aggregate (1.4.5)

(3) From the AppEngine Dashboard, on the Admin Settings page, click to
Disable your Application, wait 6 minutes, then click to Enable it. This
forces all the instances to stop and restart.
One of your instances may be in a bad state. Waiting 6 minutes should
reset the publisher retry interval.

(4) If that does not slow the consumption of quota, log in, go to Site
Admin / Preferences, and check the slow background publishing checkbox.
This will publish only as long as there are incoming submissions arriving
at the server.

(5) Check your server logs (AppEngine Dashboard, Logs). Look for errors. If
you have a malformed submission, you will need to manually repair the
submission. The malformed submission would block the publisher from
publishing any data submitted after the problematic data record.

···

On Fri, Feb 6, 2015 at 10:48 AM, John Berini wrote:

Hi There,

I'm running into the same issue that Greg ran into last year. In mid
January, I tried exporting all my uploaded data from ODK Aggregate to a
google fusion table, and I've been having major issues ever since. My
Datastore Read Operations quota on Google's AppEngine has been maxing out
everyday for roughly the last month, and as a result I haven't been able to
login to ODK Aggregate. In the last 24 hours AppEngine has logged 317
errors due to "Upload Submissions Tasks", and this has been occurring daily
since the time I tried publishing my data to a Google Fusion Table (January
14th). I tried completely disabling the app on Jan 22, and I've also
completely deleted the Fusion Table in an attempt to discontinue any
looping tasks that might be associated with the table.

Nothing that I've tried appears to be working. Any assistance anyone can
offer would be greatly appreciated.

John

On Friday, August 9, 2013 at 4:20:24 PM UTC-5, Greg wrote:
Done! Thank you for the warning. Sometimes it can be difficult keeping
track of Google's updates.

On Wed, Aug 7, 2013 at 10:49 AM, Mitch Sundt mitche...@gmail.com wrote:
Great. Glad it resolved the issue.

Since you are using AppEngine, you should upgrade to the ODK Aggregate
1.3.3 when you have a chance.

Google has expanded the length of their unique keys and this may cause
problems with publishers should Google start giving the longer unique keys
to your submissions. If you again start having publishers in the paused
state, this will be the first thing to try.

Mitch

On Wed, Aug 7, 2013 at 8:48 AM, Greg greg....@edmonton.ca wrote:
 Just as a followup.  I deleted the paused publishers in ODK and
re-enabled the Faster Background Actions. After about 24 hours GAE seems
to be acting normally again and not racking up charges. FYI I'm guessing
this started when Google had a "problem" around June 10 & 11 and then I had
to republish everything.
 Anyway, thank you for your help Mitch!  I greatly appreciate it.
Greg

On Friday, August 2, 2013 3:30:53 PM UTC-6, Greg wrote:
Hi,
No I didn't see that, it's a shared account so thank you very much
for letting me know. I'm just leaving for the weekend but I suspect you
probably found the issue. I didn't know ODK would retain publishing links
like that. I'm assuming it does that after a form upgrade? Or would that
be after deleting a form and uploading a new one of the same name and ID.
In any case I'll have to check when I changed the form and delete the
PAUSED ones.

Thank you very much again!

On Friday, August 2, 2013 1:34:08 PM UTC-6, Mitch Sundt wrote:
Not sure if you saw this e-mail I sent to the edmptestlab@ gmail
account:
---start--------------------
Hi,

A number of your publishers are in the 'PAUSED' state which indicates
either a transient or a ongoing failure to publish to the destination
servers.

In one instance, for this publisher:

Form: Check-In Form
To: Fusion Tables
Establishment Date: 2013-06-05 21:26:24.969000
Mode: Both Uploading and Streaming

The error is that the fusion table no longer exists.

You should double-check that the links to the fusion tables and
spreadsheets that are in the 'PAUSED' state actually work. If they don't,
you should delete those publishers.

Otherwise, ODK Aggregate will keep attempting to publish to those
locations, fail, and re-attempt it a short time later. Right now, that is
every 90 seconds. In a future release we will retry once 90 seconds later,
then retry only every 7.5 minutes, or every 15 minutes if the server is not
in use.

--end--------------------

w.r.t. keeping the box checked -- this simply sets the behavior back
to the highly conservative pre-1.3.1 publishing rate (a much slower rate,
aimed at conserving quota for interactive use, and trickling data into
Fusion Tables, etc.). Once the 7.5 minute back-off is available
(mentioned in the above e-mail), you should not need to check this
checkbox, as the server will then automatically shift failing publishers
into a highly-conservative publishing-retry cycle, and move them back into
the aggressive publishing cycle once they stop failing.

Mitch

On Fri, Aug 2, 2013 at 10:06 AM, Greg greg....@edmonton.ca wrote:
Hi Mitch,
 I checked the *'D**isable faster background actions (exports,
publishing, form deletion) (slows quota usage on Google AppEngine)'*
and that has helped immensely. The logs are still full of the ":
Time left on lock:" entries but the Errors have mostly disappeared and we
aren't getting a constant (and expensive) /gae/uploadSubmissionsTask load.
 I wasn't even aware of the validation rules in Fusion and
Spreadsheets. That function wasn't available from the "Change Variables"
section of Fusion Tables when I last set these up in June. I checked
through anyway and no validation rules are in place. However I do have
calculated columns in place. Like I said though, from the few records I
was able to glean from errors, the ones it keeps trying to insert and
getting errors on, are the same ones I've made manual changes to in Fusion
Tables. The spreadsheets also return errors but nothing to indicate which
entries are resulting in errors. Admittedly though I'm not the best at
sorting through the logs and mostly I just keep looking through until my
eyeballs bleed. I don't seem to understand the flavour of regex used.
So as for deleting offending records, perhaps I can just delete
all the records of the offending tables since we use Fusion Tables for our
data. As for the "Disable faster background actions", what is the long
term consequence of keeping that box checked? I feel a little nervous
about keeping a stop gap measure in place without curing the initial source
of the problem. I'll have to contact the Fusion Tables team too but I
wonder if they will be like the Google App Engine team that basically said
Google's end is good, it's really my problem.
 I've added you as a developer to App Engine in case you want to
snoop around and see what's wrong or research for ODK development. Thanks
again for the help!

On Thursday, August 1, 2013 4:16:44 PM UTC-6, Mitch Sundt wrote:
Argh!

My untested suspicion is that you may have set up a validation rule
or constraint, and, when Aggregate tries to submit a record that violates
that constraint, Fusion Tables rejects the publishing of that submission,
causing the publisher to stall and never get past the failing record (it
sits in an infinite retry).

If that is what is happening, the two choices are:

(1) Log onto ODK Aggregate as a user with Forms Management
privileges.
(2) find the offending records on the submissions page (e.g., via
filtering)
(3) click the red 'X' to delete those records (permanently) from
ODK Aggregate

Or,

Eliminate the validation constraints in the Fusion Table or
Spreadsheet that you are publishing to.

Mitch

On Thu, Aug 1, 2013 at 3:13 PM, Mitch Sundt mitche...@gmail.com wrote:
As a stop-gap,

log onto ODK Aggregate
Go to Site Admin / Preferences
Check the checkbox labelled

*'Disable faster background actions (exports, publishing, form
deletion) (slows quota usage on Google AppEngine)' *
This should restore your usage to prior values (it disables faster
publishing, restoring behavior to the pre-1.3.1 behavior).

Do you have any validation rules set up on your fusion tables or
spreadsheets?
In the last example, 'Backend error' would indicate that it is an
error on the Fusion Tables side. Have you contacted the Fusion Tables team?

Do you have any validation rules set up on your tables or
spreadsheets?

On Thu, Aug 1, 2013 at 12:35 PM, Greg greg....@edmonton.ca wrote:
Hi,
I'm running ODK Aggregate v.1.3.2 on Google App Engine and the
frontend-queue is running 24/7 with tasks and racking up some big charges.
Most of this time we are not using Aggregate. Google Enterprise Support
confirmed that most of the load came from /gae/uploadSubmissionsTask &
/gae/watchdog but claims everything is running normally on their end.
I'm dug through the server logs and most of the Info type of
entries are the

I 2013-08-01 13:02:30.447 [s~edmpestlab/1.369201346591658058].:
Time left on lock: -12380028.
 These Warning messages I hadn't seen before today:
W 2013-08-01 13:02:50.717 org.opendatakit.aggregate.task.UploadSubmissionsWorkerImpl
uploadAllSubmissions: Upload invoked when operational status is not ACTIVE

W 2013-08-01 13:02:50.953 org.opendatakit.aggregate.task.UploadSubmissionsWorkerImpl
uploadAllSubmissions: Unable to acquire lock
Examples of the error messages:
E 2013-08-01 13:03:08.889 org.opendatakit.aggregate.exte
rnalservice.GoogleSpreadsheet insertData: Unable to insert data
into spreadsheet exception: OK

E 2013-08-01 13:03:09.236 org.opendatakit.aggregate.task
.gae.servlet.UploadSubmissionsTaskServlet doGet:
org.opendatakit.aggregate.exception.ODKExternalServiceException:
com.go

E 2013-08-01 13:01:54.411 org.opendatakit.aggregate.task
.gae.servlet.UploadSubmissionsTaskServlet doGet:
org.opendatakit.aggregate.exception.ODKExternalServiceException:
{ "err

E 2013-08-01 13:03:07.456
org.opendatakit.aggregate.task.gae.servlet.UploadSubmissionsTaskServlet
doGet: org.opendatakit.aggregate.exception.
ODKExternalServiceException: { "error": { "errors": [ {
"domain": "global", "reason": "backendError", "message": "Backend
Error" } ], "code": 503, "message": "Backend Error" }}INSERT INTO
1IQS48A9Tq-R8s6geNV3yQvH4MPbaJCqsrX1861I
('meta-instance-id','meta-model-version','meta-ui-versi
on','meta-submission-date','meta-is-complete','meta-da
te-marked-as-complete','FormVersion','DateAuto','deviceid',
'simid','PhoneNumber','Crew','PhoneUserLab','PhoneUserOps','
PhoneUserOth','PhoneUser','Phonename') VALUES
('uuid:ef0b5fd9-2ebd-487c-b451-dfeda5dd13a6','2013060402','','06/26/2013
16:04:44.807','true','06/26/2013 16:04:44.807','ver_2013-Jun-04_rev02','06/26/2013
16:04:09.785','356854040053167','8912230000040758245','','Pe
stLab','BillSmith','','','BillSmith','Myphone bill smith') ;
INSERT INTO 1IQS48A9Tq-R8s6geNV3yQvH4MPbaJCqsrX1861I
('meta-instance-id','meta-model-version','meta-ui-versi
on','meta-submission-date','meta-is-complete','meta-da
te-marked-as-complete','FormVersion','DateAuto','deviceid',
'simid','PhoneNumber','Crew','PhoneUserLab','PhoneUserOps','
PhoneUserOth','PhoneUser','Phonename') VALUES
('uuid:1db681c8-2716-4d7c-829a-45195b45955c','2013060402','','08/01/2013
18:41:53.837','true','08/01/2013 18:41:53.837','ver_2013-Jun-04_rev02','08/01/2013
18:41:34.003','356854040053100','8912230000127052512','17809
185555','PestLab','JohnSmith','','','JohnSmith','Mine')
 The last error message in the list is the shortest example
of an error that happens for more than one form. I compared the data in
these messages with existing data in the corresponding Fusion Tables and
it's always for records that I had to directly alter in the Fusion Tables.
 Purging the waiting tasks in the frontend-queue stops
activity in the Dashboard charts briefly and pausing the frontend-queue
stops most activity for as long as it is paused while still queuing tasks.
256 tasks piled up last night. I was running version 1.3.1 and then
upgraded to 1.3.2 in hopes that might solve the problem but nothing changed.
 To me it seems like something is "stuck" and keeps
resubmitting but I really don't know how to fix it. I'm a little cautious
of pushing too many buttons without knowing what things like "Flushing
Memcache" etc. actually do. I've searched through ODK Community and Google
App Engine groups but am not finding anything similar.
 Any suggestions of what to do, information needed to solve
the problem, places to refer my question, wild-ass guesses would be greatly
appreciated. As a side question, does anyone know of good documentation on
App Engine from an administrators point of view? All the documentation
I've seen seems oriented to programming.

Thank you.

--

Post: opend...@googlegroups.com
Unsubscribe: opendatakit...@googlegroups.com

Options: http://groups.google.com/group/opendatakit?hl=en

You received this message because you are subscribed to the
Google Groups "ODK Community" group.
To unsubscribe from this group and stop receiving emails from it,
send an email to opendatakit...@googlegroups.com.

For more options, visit https://groups.google.com/groups/opt_out.
--
Mitch Sundt
Software Engineer
University of Washington
mitche...@gmail.com
--
Mitch Sundt
Software Engineer
University of Washington
mitche...@gmail.com
--

Post: opend...@googlegroups.com
Unsubscribe: opendatakit...@googlegroups.com
Options: http://groups.google.com/group/opendatakit?hl=en

You received this message because you are subscribed to the Google
Groups "ODK Community" group.
To unsubscribe from this group and stop receiving emails from it,
send an email to opendatakit...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.
--
Mitch Sundt
Software Engineer
University of Washington
mitche...@gmail.com
--
--
Post: opend...@googlegroups.com
Unsubscribe: opendatakit...@googlegroups.com
Options: http://groups.google.com/group/opendatakit?hl=en
You received this message because you are subscribed to the Google
Groups "ODK Community" group.
To unsubscribe from this group and stop receiving emails from it, send
an email to opendatakit...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.
--
Mitch Sundt
Software Engineer
University of Washington
mitche...@gmail.com

--

Post: opend...@googlegroups.com
Unsubscribe: opendatakit...@googlegroups.com
Options: http://groups.google.com/group/opendatakit?hl=en

You received this message because you are subscribed to a topic in the
Google Groups "ODK Community" group.
To unsubscribe from this topic, visit https://groups.google.com/d/
topic/opendatakit/MBzQgOHKVNc/unsubscribe.
To unsubscribe from this group and all its topics, send an email to
opendatakit...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.
--
Greg Pommen
Biological Sciences Technical Assistant
City of Edmonton Environmental Services
12304-107 St.
Edmonton, AB T5G 2S7
(780) 496-6906
fax: (780) 496-4978
greg....@edmonton.ca
--

Post: opendatakit@googlegroups.com
Unsubscribe: opendatakit+unsubscribe@googlegroups.com
Options: http://groups.google.com/group/opendatakit?hl=en

You received this message because you are subscribed to the Google Groups
"ODK Community" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to opendatakit+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--
Mitch Sundt
Software Engineer
University of Washington
mitchellsundt@gmail.com

John_Berini · February 6, 2015, 7:38pm

Thanks for the prompt reply Mitch. I have a few questions/points for the
sake of clarification. The numbering refers to your suggestions below.

First a question:

To "delete publisher" do I need to accomplish this within Google
Drive or within the AppsEngine? I've removed all but my personal
ownership and developer accounts from appengine.

Now, for a couple points for clarification:

I'm not using ODK Aggregate on my machine, but rather I'm doing all
this through appspot after pushing data to the cloud directly from the
tablet we used to collect the data on.
For some godforsaken reason I'm unable to access my "Application
Settings" under "Administration" from the AppEngine dashboard. It's been
like this for the last few weeks. When I click on the "Application
Settings" link I get a "Server Error" message. I have another test
account that I use and can access those settings under that account.
See #3.
I do have a few recurring error:

          W2015-02-06
          11:32:10.841org.springframework.security.config.http.MatcherType
          fromElement: 'path-type' is deprecated. Please use
          'request-matcher' instead.

          W2015-02-06
          11:32:10.841org.springframework.security.config.http.MatcherType
          fromElement: 'path-type' is deprecated. Please use
          'request-matcher' instead.

...and then a few lines later:

          E2015-02-06
          11:32:11.152org.springframework.web.context.ContextLoader
          initWebApplicationContext: Context initialization failed
          org.springframework.beans.factory.BeanCreationE

          W2015-02-06 11:32:11.153Failed startup of context
          com.google.apphosting.utils.jetty.RuntimeAppEngineWebAppContext@11a5636{/,/base/data/home/apps/s~jberini/1.38152020947335098

          W2015-02-06 11:32:11.154Nested in
          org.springframework.beans.factory.BeanCreationException:
          Error creating bean with name
          'hierarchicalRoleRelationships' defined in ServletCon

          C2015-02-06 11:32:11.159Uncaught exception from servlet
          org.springframework.beans.factory.BeanCreationException:
          Error creating bean with name 'hierarchicalRoleRelationships'

It's this last line above ("Uncaught exception from servelet") where
things keep getting caught up. How do I manually repair (or delete) this
submission if I can't access the data that are causing the problem?

John

···

On 2/6/15 1:03 PM, Mitch Sundt wrote: > Did you delete the publisher? > > This can occur if you delete the fusion table, but not the publisher. > > It can also happen if a submission is corrupted -- you would notice > this by the publisher never publishing new data into your fusion table. > > Things to try: > > (1) if you deleted the fusion table, make sure you deleted the > publisher into it. > > (2) upgrade to the latest version of ODK Aggregate (1.4.5) > > (3) From the AppEngine Dashboard, on the Admin Settings page, click to > Disable your Application, wait 6 minutes, then click to Enable it. > This forces all the instances to stop and restart. > One of your instances may be in a bad state. Waiting 6 minutes should > reset the publisher retry interval. > > (4) If that does not slow the consumption of quota, log in, go to Site > Admin / Preferences, and check the slow background publishing > checkbox. This will publish only as long as there are incoming > submissions arriving at the server. > > (5) Check your server logs (AppEngine Dashboard, Logs). Look for > errors. If you have a malformed submission, you will need to manually > repair the submission. The malformed submission would block the > publisher from publishing any data submitted after the problematic > data record. > > > On Fri, Feb 6, 2015 at 10:48 AM, John Berini <jberini@gmail.com > wrote: > > Hi There, > > I'm running into the same issue that Greg ran into last year. In > mid January, I tried exporting all my uploaded data from ODK > Aggregate to a google fusion table, and I've been having major > issues ever since. My Datastore Read Operations quota on Google's > AppEngine has been maxing out everyday for roughly the last month, > and as a result I haven't been able to login to ODK Aggregate. In > the last 24 hours AppEngine has logged 317 errors due to "Upload > Submissions Tasks", and this has been occurring daily since the > time I tried publishing my data to a Google Fusion Table (January > 14th). I tried completely disabling the app on Jan 22, and I've > also completely deleted the Fusion Table in an attempt to > discontinue any looping tasks that might be associated with the > table. > > Nothing that I've tried appears to be working. Any assistance > anyone can offer would be greatly appreciated. > > John > > > On Friday, August 9, 2013 at 4:20:24 PM UTC-5, Greg wrote: > > Done! Thank you for the warning. Sometimes it can be > difficult keeping track of Google's updates. > > > On Wed, Aug 7, 2013 at 10:49 AM, Mitch Sundt wrote: > > Great. Glad it resolved the issue. > > Since you are using AppEngine, you should upgrade to the > ODK Aggregate 1.3.3 when you have a chance. > > Google has expanded the length of their unique keys and > this may cause problems with publishers should Google > start giving the longer unique keys to your submissions. > If you again start having publishers in the paused state, > this will be the first thing to try. > > Mitch > > > > On Wed, Aug 7, 2013 at 8:48 AM, Greg wrote: > > Just as a followup. I deleted the paused > publishers in ODK and re-enabled the Faster Background > Actions. After about 24 hours GAE seems to be acting > normally again and not racking up charges. FYI I'm > guessing this started when Google had a "problem" > around June 10 & 11 and then I had to republish > everything. > > Anyway, thank you for your help Mitch! I greatly > appreciate it. > > Greg > > > On Friday, August 2, 2013 3:30:53 PM UTC-6, Greg wrote: > > Hi, > No I didn't see that, it's a shared account so > thank you very much for letting me know. I'm just > leaving for the weekend but I suspect you probably > found the issue. I didn't know ODK would retain > publishing links like that. I'm assuming it does > that after a form upgrade? Or would that be after > deleting a form and uploading a new one of the > same name and ID. In any case I'll have to check > when I changed the form and delete the PAUSED ones. > > Thank you very much again! > > > On Friday, August 2, 2013 1:34:08 PM UTC-6, Mitch Sundt wrote: > > Not sure if you saw this e-mail I sent to the > edmptestlab@ gmail account: > ---start-------------------- > Hi, > > A number of your publishers are in the > 'PAUSED' state which indicates either a > transient or a ongoing failure to publish to > the destination servers. > > In one instance, for this publisher: > > Form: Check-In Form > To: Fusion Tables > Establishment Date: 2013-06-05 21:26:24.969000 > Mode: Both Uploading and Streaming > > The error is that the fusion table no longer > exists. > > You should double-check that the links to the > fusion tables and spreadsheets that are in the > 'PAUSED' state actually work. If they don't, > you should delete those publishers. > > Otherwise, ODK Aggregate will keep attempting > to publish to those locations, fail, and > re-attempt it a short time later. Right now, > that is every 90 seconds. In a future release > we will retry once 90 seconds later, then > retry only every 7.5 minutes, or every 15 > minutes if the server is not in use. > > --end-------------------- > > w.r.t. keeping the box checked -- this simply > sets the behavior back to the highly > conservative pre-1.3.1 publishing rate (a much > slower rate, aimed at conserving quota for > interactive use, and trickling data into > Fusion Tables, etc.). Once the 7.5 minute > back-off is available (mentioned in the above > e-mail), you should not need to check this > checkbox, as the server will then > automatically shift failing publishers into a > highly-conservative publishing-retry cycle, > and move them back into the aggressive > publishing cycle once they stop failing. > > Mitch > > On Fri, Aug 2, 2013 at 10:06 AM, Greg wrote: > > Hi Mitch, > > I checked the *'D**isable faster > background actions (exports, publishing, > form deletion) (slows quota usage on > Google AppEngine)'* and that has helped > immensely. The logs are still full of the > ": Time left on lock:" entries but > the Errors have mostly disappeared and we > aren't getting a constant (and expensive) > /gae/uploadSubmissionsTask load. > > I wasn't even aware of the validation > rules in Fusion and Spreadsheets. That > function wasn't available from the "Change > Variables" section of Fusion Tables when I > last set these up in June. I checked > through anyway and no validation rules are > in place. However I do have calculated > columns in place. Like I said though, > from the few records I was able to glean > from errors, the ones it keeps trying to > insert and getting errors on, are the same > ones I've made manual changes to in Fusion > Tables. The spreadsheets also return > errors but nothing to indicate which > entries are resulting in > errors. Admittedly though I'm not the > best at sorting through the logs and > mostly I just keep looking through until > my eyeballs bleed. I don't seem to > understand the flavour of regex used. > > So as for deleting offending records, > perhaps I can just delete all the records > of the offending tables since we use > Fusion Tables for our data. As for the > "Disable faster background actions", what > is the long term consequence of keeping > that box checked? I feel a little nervous > about keeping a stop gap measure in place > without curing the initial source of the > problem. I'll have to contact the Fusion > Tables team too but I wonder if they will > be like the Google App Engine team that > basically said Google's end is good, it's > really my problem. > > I've added you as a developer to App > Engine in case you want to snoop around > and see what's wrong or research for ODK > development. Thanks again for the help! > > > > > > On Thursday, August 1, 2013 4:16:44 PM UTC-6, Mitch Sundt wrote: > > Argh! > > My untested suspicion is that you may > have set up a validation rule or > constraint, and, when Aggregate tries > to submit a record that violates that > constraint, Fusion Tables rejects the > publishing of that submission, causing > the publisher to stall and never get > past the failing record (it sits in an > infinite retry). > > If that is what is happening, the two > choices are: > > (1) Log onto ODK Aggregate as a user > with Forms Management privileges. > (2) find the offending records on the > submissions page (e.g., via filtering) > (3) click the red 'X' to delete those > records (permanently) from ODK Aggregate > > Or, > > Eliminate the validation constraints > in the Fusion Table or Spreadsheet > that you are publishing to. > > Mitch > > > > On Thu, Aug 1, 2013 at 3:13 PM, Mitch Sundt wrote: > > As a stop-gap, > > log onto ODK Aggregate > Go to Site Admin / Preferences > Check the checkbox labelled > *'Disable faster background > actions (exports, publishing, form > deletion) (slows quota usage on > Google AppEngine)' > > * > This should restore your usage to > prior values (it disables faster > publishing, restoring behavior to > the pre-1.3.1 behavior). > > Do you have any validation rules > set up on your fusion tables or > spreadsheets? > In the last example, 'Backend > error' would indicate that it is > an error on the Fusion Tables > side. Have you contacted the > Fusion Tables team? > > Do you have any validation rules > set up on your tables or spreadsheets? > > * > * > > > On Thu, Aug 1, 2013 at 12:35 PM, Greg wrote: > > Hi, > I'm running ODK Aggregate > v.1.3.2 on Google App Engine > and the frontend-queue is > running 24/7 with tasks and > racking up some big charges. > Most of this time we are not > using Aggregate. Google > Enterprise Support confirmed > that most of the load came > from > /gae/uploadSubmissionsTask & > /gae/watchdog but claims > everything is running normally > on their end. > > > I'm dug through the server > logs and most of the Info type > of entries are the > > I 2013-08-01 13:02:30.447 > [s~edmpestlab/1.369201346591658058].: > Time left on lock: -12380028. > > > These Warning messages I > hadn't seen before today: > > W 2013-08-01 13:02:50.717 > org.opendatakit.aggregate.task.UploadSubmissionsWorkerImpl > uploadAllSubmissions: Upload > invoked when operational > status is not ACTIVE > > W 2013-08-01 13:02:50.953 > org.opendatakit.aggregate.task.UploadSubmissionsWorkerImpl > uploadAllSubmissions: Unable > to acquire lock > > > Examples of the error > messages: > > E 2013-08-01 13:03:08.889 > org.opendatakit.aggregate.externalservice.GoogleSpreadsheet > insertData: Unable to insert > data into spreadsheet of various spreadsheets> > exception: OK > > E 2013-08-01 13:03:09.236 > org.opendatakit.aggregate.task.gae.servlet.UploadSubmissionsTaskServlet > doGet: > org.opendatakit.aggregate.exception.ODKExternalServiceException: > com.go > > E 2013-08-01 13:01:54.411 > org.opendatakit.aggregate.task.gae.servlet.UploadSubmissionsTaskServlet > doGet: > org.opendatakit.aggregate.exception.ODKExternalServiceException: > { "err > > E 2013-08-01 13:03:07.456 > org.opendatakit.aggregate.task.gae.servlet.UploadSubmissionsTaskServlet > doGet: > org.opendatakit.aggregate.exception.ODKExternalServiceException: > { "error": { "errors": [ { > "domain": "global", > "reason": "backendError", > "message": "Backend Error" > } ], "code": 503, > "message": "Backend Error" > }}INSERT INTO > 1IQS48A9Tq-R8s6geNV3yQvH4MPbaJCqsrX1861I > ('*meta-instance-id*','*meta-model-version*','*meta-ui-version*','*meta-submission-date*','*meta-is-complete*','*meta-date-marked-as-complete*','FormVersion','DateAuto','deviceid','simid','PhoneNumber','Crew','PhoneUserLab','PhoneUserOps','PhoneUserOth','PhoneUser','Phonename') > VALUES > ('uuid:ef0b5fd9-2ebd-487c-b451-dfeda5dd13a6','2013060402','','06/26/2013 > 16:04:44.807','true','06/26/2013 > 16:04:44.807','ver_2013-Jun-04_rev02','06/26/2013 > 16:04:09.785','356854040053167','8912230000040758245','','PestLab','BillSmith','','','BillSmith','Myphone > bill smith') ; INSERT INTO > 1IQS48A9Tq-R8s6geNV3yQvH4MPbaJCqsrX1861I > ('*meta-instance-id*','*meta-model-version*','*meta-ui-version*','*meta-submission-date*','*meta-is-complete*','*meta-date-marked-as-complete*','FormVersion','DateAuto','deviceid','simid','PhoneNumber','Crew','PhoneUserLab','PhoneUserOps','PhoneUserOth','PhoneUser','Phonename') > VALUES > ('uuid:1db681c8-2716-4d7c-829a-45195b45955c','2013060402','','08/01/2013 > 18:41:53.837','true','08/01/2013 > 18:41:53.837','ver_2013-Jun-04_rev02','08/01/2013 > 18:41:34.003','356854040053100','8912230000127052512','17809185555','PestLab','JohnSmith','','','JohnSmith','Mine') > > > > The last error message in > the list is the shortest > example of an error that > happens for more than one > form. I compared the data in > these messages with existing > data in the > corresponding Fusion > Tables and it's always for > records that I had to directly > alter in the Fusion Tables. > > Purging the waiting tasks in > the frontend-queue stops > activity in the Dashboard > charts briefly and pausing the > frontend-queue stops most > activity for as long as it is > paused while still queuing > tasks. 256 tasks piled up > last night. I was running > version 1.3.1 and then > upgraded to 1.3.2 in hopes > that might solve the problem > but nothing changed. > > To me it seems like > something is "stuck" and keeps > resubmitting but I really > don't know how to fix it. I'm > a little cautious of pushing > too many buttons without > knowing what things like > "Flushing Memcache" etc. > actually do. I've searched > through ODK Community and > Google App Engine groups but > am not finding anything similar. > > Any suggestions of what > to do, information needed to > solve the problem, places to > refer my question, wild-ass > guesses would be greatly > appreciated. As a side > question, does anyone know of > good documentation on App > Engine from an administrators > point of view? All the > documentation I've seen seems > oriented to programming. > > Thank you. > > > > -- > -- > Post: opend...@googlegroups.com > Unsubscribe: > opendatakit...@googlegroups.com > > Options: > http://groups.google.com/group/opendatakit?hl=en > > --- > You received this message > because you are subscribed to > the Google Groups "ODK > Community" group. > To unsubscribe from this group > and stop receiving emails from > it, send an email to > opendatakit...@googlegroups.com. > > For more options, visit > https://groups.google.com/groups/opt_out. > > > > > > -- > Mitch Sundt > Software Engineer > University of Washington > mitche...@gmail.com > > > > > -- > Mitch Sundt > Software Engineer > University of Washington > mitche...@gmail.com > > -- > -- > Post: opend...@googlegroups.com > Unsubscribe: opendatakit...@googlegroups.com > Options: > http://groups.google.com/group/opendatakit?hl=en > > --- > You received this message because you are > subscribed to the Google Groups "ODK > Community" group. > To unsubscribe from this group and stop > receiving emails from it, send an email to > opendatakit...@googlegroups.com. > For more options, visit > https://groups.google.com/groups/opt_out. > > > > > > -- > Mitch Sundt > Software Engineer > University of Washington > mitche...@gmail.com > > -- > -- > Post: opend...@googlegroups.com > Unsubscribe: opendatakit...@googlegroups.com > Options: http://groups.google.com/group/opendatakit?hl=en > > --- > You received this message because you are subscribed > to the Google Groups "ODK Community" group. > To unsubscribe from this group and stop receiving > emails from it, send an email to > opendatakit...@googlegroups.com. > For more options, visit > https://groups.google.com/groups/opt_out. > > > > > > -- > Mitch Sundt > Software Engineer > University of Washington > mitche...@gmail.com > -- > -- > Post: opend...@googlegroups.com > Unsubscribe: opendatakit...@googlegroups.com > Options: http://groups.google.com/group/opendatakit?hl=en > > --- > You received this message because you are subscribed to a > topic in the Google Groups "ODK Community" group. > To unsubscribe from this topic, visit > https://groups.google.com/d/topic/opendatakit/MBzQgOHKVNc/unsubscribe. > To unsubscribe from this group and all its topics, send an > email to opendatakit...@googlegroups.com. > For more options, visit > https://groups.google.com/groups/opt_out. > > > > > > -- > Greg Pommen > Biological Sciences Technical Assistant > City of Edmonton Environmental Services > 12304-107 St. > Edmonton, AB T5G 2S7 > (780) 496-6906 > fax: (780) 496-4978 > greg....@edmonton.ca > > -- > -- > Post: opendatakit@googlegroups.com > > Unsubscribe: opendatakit+unsubscribe@googlegroups.com > > Options: http://groups.google.com/group/opendatakit?hl=en > > --- > You received this message because you are subscribed to the Google > Groups "ODK Community" group. > To unsubscribe from this group and stop receiving emails from it, > send an email to opendatakit+unsubscribe@googlegroups.com > . > For more options, visit https://groups.google.com/d/optout. > > > > > -- > Mitch Sundt > Software Engineer > University of Washington > mitchellsundt@gmail.com > -- > -- > Post: opendatakit@googlegroups.com > Unsubscribe: opendatakit+unsubscribe@googlegroups.com > Options: http://groups.google.com/group/opendatakit?hl=en > > --- > You received this message because you are subscribed to the Google > Groups "ODK Community" group. > To unsubscribe from this group and stop receiving emails from it, send > an email to opendatakit+unsubscribe@googlegroups.com > . > For more options, visit https://groups.google.com/d/optout.

Mitch_S · February 6, 2015, 8:50pm

(1) yes, this is done on ODK Aggregate, Forms Management / Published Data
tab.

(2) correct, you are running on appengine.

(3) Google breaks their sites on a fairly regular basis w.r.t. using
multiple gmail accounts (I have 6 accounts). When the AppEngine site is
broken, you generally need to sign out of all your Google accounts (Gmail
and App Domain), then sign into the one account that has access to the
AppEngine Dashboard. Sometimes you can use a private browser window and
avoid logging out, sometimes you can't. This breakage happens on at least
one of their sites about every 2 weeks and lasts for about 3 days.

(5) The first 2 (W) warnings can be ignored. The (E) error and (S) severe
lines are significant.

Are you currently out of quota?

That would explain those. They would occur if you are out of quota - the
application is accessing the datastore to retrieve the privileges of all
the users registered in the Site Admin / Permissions page; if you are out
of quota, you will see failures at this point.

If you are not out of quota, they would indicate a problem with your last
upload of ODK Aggregate to the server. Try re-running the upload script
(click on the HTA file) under the directory where the installer placed its
files. You don't need to re-run the installer, just re-run the upload
script.

Since quota resets at midnight PST, if that is an inconvenient time, once
you get past (3) and can log in, you can Disable your application, and
leave it disabled until after the quota resets. You can then enable the
application and log in and work (until you run out of quota).

···

On Fri, Feb 6, 2015 at 11:38 AM, John Berini wrote:

Thanks for the prompt reply Mitch. I have a few questions/points for the
sake of clarification. The numbering refers to your suggestions below.

First a question:

To "delete publisher" do I need to accomplish this within Google Drive
or within the AppsEngine? I've removed all but my personal ownership and
developer accounts from appengine.

Now, for a couple points for clarification:

I'm not using ODK Aggregate on my machine, but rather I'm doing all
this through appspot after pushing data to the cloud directly from the
tablet we used to collect the data on.

For some godforsaken reason I'm unable to access my "Application
Settings" under "Administration" from the AppEngine dashboard. It's been
like this for the last few weeks. When I click on the "Application
Settings" link I get a "Server Error" message. I have another test account
that I use and can access those settings under that account.

See #3.

I do have a few recurring error:

W2015-02-06 11:32:10.841 org.springframework.security.config.http.MatcherType
fromElement: 'path-type' is deprecated. Please use 'request-matcher'
instead.

W2015-02-06 11:32:10.841 org.springframework.security.config.http.MatcherType
fromElement: 'path-type' is deprecated. Please use 'request-matcher'
instead.

...and then a few lines later:

E2015-02-06 11:32:11.152 org.springframework.web.context.ContextLoader
initWebApplicationContext: Context initialization failed
org.springframework.beans.factory.BeanCreationE

W2015-02-06 11:32:11.153 Failed startup of context
com.google.apphosting.utils.jetty.RuntimeAppEngineWebAppContext@11a5636
{/,/base/data/home/apps/s~jberini/1.38152020947335098

W2015-02-06 11:32:11.154 Nested in
org.springframework.beans.factory.BeanCreationException: Error creating
bean with name 'hierarchicalRoleRelationships' defined in ServletCon

C2015-02-06 11:32:11.159 Uncaught exception from servlet
org.springframework.beans.factory.BeanCreationException: Error creating
bean with name 'hierarchicalRoleRelationships'

It's this last line above ("Uncaught exception from servelet") where
things keep getting caught up. How do I manually repair (or delete) this
submission if I can't access the data that are causing the problem?

John

On 2/6/15 1:03 PM, Mitch Sundt wrote:

Did you delete the publisher?

This can occur if you delete the fusion table, but not the publisher.

It can also happen if a submission is corrupted -- you would notice this
by the publisher never publishing new data into your fusion table.

Things to try:

(1) if you deleted the fusion table, make sure you deleted the publisher
into it.

(2) upgrade to the latest version of ODK Aggregate (1.4.5)

(3) From the AppEngine Dashboard, on the Admin Settings page, click to
Disable your Application, wait 6 minutes, then click to Enable it. This
forces all the instances to stop and restart.
One of your instances may be in a bad state. Waiting 6 minutes should
reset the publisher retry interval.

(4) If that does not slow the consumption of quota, log in, go to Site
Admin / Preferences, and check the slow background publishing checkbox.
This will publish only as long as there are incoming submissions arriving
at the server.

(5) Check your server logs (AppEngine Dashboard, Logs). Look for errors.
If you have a malformed submission, you will need to manually repair the
submission. The malformed submission would block the publisher from
publishing any data submitted after the problematic data record.

On Fri, Feb 6, 2015 at 10:48 AM, John Berini jberini@gmail.com wrote:
Hi There,

I'm running into the same issue that Greg ran into last year. In mid
January, I tried exporting all my uploaded data from ODK Aggregate to a
google fusion table, and I've been having major issues ever since. My
Datastore Read Operations quota on Google's AppEngine has been maxing out
everyday for roughly the last month, and as a result I haven't been able to
login to ODK Aggregate. In the last 24 hours AppEngine has logged 317
errors due to "Upload Submissions Tasks", and this has been occurring daily
since the time I tried publishing my data to a Google Fusion Table (January
14th). I tried completely disabling the app on Jan 22, and I've also
completely deleted the Fusion Table in an attempt to discontinue any
looping tasks that might be associated with the table.

Nothing that I've tried appears to be working. Any assistance anyone
can offer would be greatly appreciated.

John

On Friday, August 9, 2013 at 4:20:24 PM UTC-5, Greg wrote:
Done! Thank you for the warning. Sometimes it can be difficult keeping
track of Google's updates.

On Wed, Aug 7, 2013 at 10:49 AM, Mitch Sundt mitche...@gmail.com wrote:
Great. Glad it resolved the issue.

Since you are using AppEngine, you should upgrade to the ODK Aggregate
1.3.3 when you have a chance.

Google has expanded the length of their unique keys and this may cause
problems with publishers should Google start giving the longer unique keys
to your submissions. If you again start having publishers in the paused
state, this will be the first thing to try.

Mitch

On Wed, Aug 7, 2013 at 8:48 AM, Greg greg....@edmonton.ca wrote:
 Just as a followup.  I deleted the paused publishers in ODK and
re-enabled the Faster Background Actions. After about 24 hours GAE seems
to be acting normally again and not racking up charges. FYI I'm guessing
this started when Google had a "problem" around June 10 & 11 and then I had
to republish everything.
  Anyway, thank you for your help Mitch!  I greatly appreciate it.
Greg

On Friday, August 2, 2013 3:30:53 PM UTC-6, Greg wrote:
Hi,
No I didn't see that, it's a shared account so thank you very
much for letting me know. I'm just leaving for the weekend but I suspect
you probably found the issue. I didn't know ODK would retain publishing
links like that. I'm assuming it does that after a form upgrade? Or would
that be after deleting a form and uploading a new one of the same name and
ID. In any case I'll have to check when I changed the form and delete the
PAUSED ones.

Thank you very much again!

On Friday, August 2, 2013 1:34:08 PM UTC-6, Mitch Sundt wrote:
Not sure if you saw this e-mail I sent to the edmptestlab@ gmail
account:
---start--------------------
Hi,

A number of your publishers are in the 'PAUSED' state which
indicates either a transient or a ongoing failure to publish to the
destination servers.

In one instance, for this publisher:

Form: Check-In Form
To: Fusion Tables
Establishment Date: 2013-06-05 21:26:24.969000
Mode: Both Uploading and Streaming

The error is that the fusion table no longer exists.

You should double-check that the links to the fusion tables and
spreadsheets that are in the 'PAUSED' state actually work. If they don't,
you should delete those publishers.

Otherwise, ODK Aggregate will keep attempting to publish to those
locations, fail, and re-attempt it a short time later. Right now, that is
every 90 seconds. In a future release we will retry once 90 seconds later,
then retry only every 7.5 minutes, or every 15 minutes if the server is not
in use.

--end--------------------

w.r.t. keeping the box checked -- this simply sets the behavior
back to the highly conservative pre-1.3.1 publishing rate (a much slower
rate, aimed at conserving quota for interactive use, and trickling data
into Fusion Tables, etc.). Once the 7.5 minute back-off is available
(mentioned in the above e-mail), you should not need to check this
checkbox, as the server will then automatically shift failing publishers
into a highly-conservative publishing-retry cycle, and move them back into
the aggressive publishing cycle once they stop failing.

Mitch

On Fri, Aug 2, 2013 at 10:06 AM, Greg greg....@edmonton.ca wrote:
Hi Mitch,
  I checked the *'D**isable faster background actions
(exports, publishing, form deletion) (slows quota usage on Google
AppEngine)'* and that has helped immensely. The logs are still
full of the ": Time left on lock:" entries but the Errors have
mostly disappeared and we aren't getting a constant (and expensive)
/gae/uploadSubmissionsTask load.
  I wasn't even aware of the validation rules in Fusion and
Spreadsheets. That function wasn't available from the "Change Variables"
section of Fusion Tables when I last set these up in June. I checked
through anyway and no validation rules are in place. However I do have
calculated columns in place. Like I said though, from the few records I
was able to glean from errors, the ones it keeps trying to insert and
getting errors on, are the same ones I've made manual changes to in Fusion
Tables. The spreadsheets also return errors but nothing to indicate which
entries are resulting in errors. Admittedly though I'm not the best at
sorting through the logs and mostly I just keep looking through until my
eyeballs bleed. I don't seem to understand the flavour of regex used.
 So as for deleting offending records, perhaps I can just
delete all the records of the offending tables since we use Fusion Tables
for our data. As for the "Disable faster background actions", what is the
long term consequence of keeping that box checked? I feel a little nervous
about keeping a stop gap measure in place without curing the initial source
of the problem. I'll have to contact the Fusion Tables team too but I
wonder if they will be like the Google App Engine team that basically said
Google's end is good, it's really my problem.
  I've added you as a developer to App Engine in case you want
to snoop around and see what's wrong or research for ODK development.
Thanks again for the help!

On Thursday, August 1, 2013 4:16:44 PM UTC-6, Mitch Sundt wrote:
Argh!

My untested suspicion is that you may have set up a validation
rule or constraint, and, when Aggregate tries to submit a record that
violates that constraint, Fusion Tables rejects the publishing of that
submission, causing the publisher to stall and never get past the failing
record (it sits in an infinite retry).

If that is what is happening, the two choices are:

(1) Log onto ODK Aggregate as a user with Forms Management
privileges.
(2) find the offending records on the submissions page (e.g.,
via filtering)
(3) click the red 'X' to delete those records (permanently) from
ODK Aggregate

Or,

Eliminate the validation constraints in the Fusion Table or
Spreadsheet that you are publishing to.

Mitch

On Thu, Aug 1, 2013 at 3:13 PM, Mitch Sundt mitche...@gmail.com wrote:
As a stop-gap,

log onto ODK Aggregate
Go to Site Admin / Preferences
Check the checkbox labelled

*'Disable faster background actions (exports, publishing, form
deletion) (slows quota usage on Google AppEngine)' *
This should restore your usage to prior values (it disables
faster publishing, restoring behavior to the pre-1.3.1 behavior).

Do you have any validation rules set up on your fusion tables
or spreadsheets?
In the last example, 'Backend error' would indicate that it is an
error on the Fusion Tables side. Have you contacted the Fusion Tables team?

Do you have any validation rules set up on your tables or
spreadsheets?

On Thu, Aug 1, 2013 at 12:35 PM, Greg greg....@edmonton.ca wrote:
Hi,
I'm running ODK Aggregate v.1.3.2 on Google App Engine and
the frontend-queue is running 24/7 with tasks and racking up some big
charges. Most of this time we are not using Aggregate. Google Enterprise
Support confirmed that most of the load came from
/gae/uploadSubmissionsTask & /gae/watchdog but claims everything is
running normally on their end.
 I'm dug through the server logs and most of the Info type
of entries are the

I 2013-08-01 13:02:30.447
[s~edmpestlab/1.369201346591658058].: Time left on lock: -12380028.
  These Warning messages I hadn't seen before today:
W 2013-08-01 13:02:50.717
org.opendatakit.aggregate.task.UploadSubmissionsWorkerImpl
uploadAllSubmissions: Upload invoked when operational status is not ACTIVE

W 2013-08-01 13:02:50.953
org.opendatakit.aggregate.task.UploadSubmissionsWorkerImpl
uploadAllSubmissions: Unable to acquire lock
 Examples of the error messages:
E 2013-08-01 13:03:08.889
org.opendatakit.aggregate.externalservice.GoogleSpreadsheet insertData:
Unable to insert data into spreadsheet
exception: OK

E 2013-08-01 13:03:09.236
org.opendatakit.aggregate.task.gae.servlet.UploadSubmissionsTaskServlet
doGet: org.opendatakit.aggregate.exception.ODKExternalServiceException:
com.go

E 2013-08-01 13:01:54.411
org.opendatakit.aggregate.task.gae.servlet.UploadSubmissionsTaskServlet
doGet: org.opendatakit.aggregate.exception.ODKExternalServiceException: {
"err

E 2013-08-01 13:03:07.456
org.opendatakit.aggregate.task.gae.servlet.UploadSubmissionsTaskServlet
doGet: org.opendatakit.aggregate.exception.ODKExternalServiceException: {
"error": { "errors": [ { "domain": "global", "reason":
"backendError", "message": "Backend Error" } ], "code": 503,
"message": "Backend Error" }}INSERT INTO
1IQS48A9Tq-R8s6geNV3yQvH4MPbaJCqsrX1861I
('meta-instance-id','meta-model-version','meta-ui-version','meta-submission-date','meta-is-complete','meta-date-marked-as-complete','FormVersion','DateAuto','deviceid','simid','PhoneNumber','Crew','PhoneUserLab','PhoneUserOps','PhoneUserOth','PhoneUser','Phonename')
VALUES ('uuid:ef0b5fd9-2ebd-487c-b451-dfeda5dd13a6','
2013060402','','06/26/2013 16:04:44.807','true','06/26/2013
16:04:44.807','ver_2013-Jun-04_rev02','06/26/2013
16:04:09.785','356854040053167','8912230000040758245','','PestLab','BillSmith','','','BillSmith','Myphone
bill smith') ; INSERT INTO 1IQS48A9Tq-R8s6geNV3yQvH4MPbaJCqsrX1861I
('meta-instance-id','meta-model-version','meta-ui-version','meta-submission-date','meta-is-complete','meta-date-marked-as-complete','FormVersion','DateAuto','deviceid','simid','PhoneNumber','Crew','PhoneUserLab','PhoneUserOps','PhoneUserOth','PhoneUser','Phonename')
VALUES ('uuid:1db681c8-2716-4d7c-829a-45195b45955c','
2013060402','','08/01/2013 18:41:53.837','true','08/01/2013
18:41:53.837','ver_2013-Jun-04_rev02','08/01/2013
18:41:34.003','356854040053100','8912230000127052512','17809185555','PestLab','JohnSmith','','','JohnSmith','Mine')
  The last error message in the list is the shortest
example of an error that happens for more than one form. I compared the
data in these messages with existing data in the corresponding Fusion
Tables and it's always for records that I had to directly alter in the
Fusion Tables.
  Purging the waiting tasks in the frontend-queue stops
activity in the Dashboard charts briefly and pausing the frontend-queue
stops most activity for as long as it is paused while still queuing tasks.
256 tasks piled up last night. I was running version 1.3.1 and then
upgraded to 1.3.2 in hopes that might solve the problem but nothing changed.
  To me it seems like something is "stuck" and keeps
resubmitting but I really don't know how to fix it. I'm a little cautious
of pushing too many buttons without knowing what things like "Flushing
Memcache" etc. actually do. I've searched through ODK Community and Google
App Engine groups but am not finding anything similar.
  Any suggestions of what to do, information needed to
solve the problem, places to refer my question, wild-ass guesses would be
greatly appreciated. As a side question, does anyone know of good
documentation on App Engine from an administrators point of view? All the
documentation I've seen seems oriented to programming.

Thank you.

--

Post: opend...@googlegroups.com
Unsubscribe: opendatakit...@googlegroups.com

Options: http://groups.google.com/group/opendatakit?hl=en

You received this message because you are subscribed to the
Google Groups "ODK Community" group.
To unsubscribe from this group and stop receiving emails from
it, send an email to opendatakit...@googlegroups.com.

For more options, visit https://groups.google.com/groups/opt_out
.
--
Mitch Sundt
Software Engineer
University of Washington
mitche...@gmail.com
--
Mitch Sundt
Software Engineer
University of Washington
mitche...@gmail.com
--

Post: opend...@googlegroups.com
Unsubscribe: opendatakit...@googlegroups.com
Options: http://groups.google.com/group/opendatakit?hl=en

You received this message because you are subscribed to the Google
Groups "ODK Community" group.
To unsubscribe from this group and stop receiving emails from it,
send an email to opendatakit...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.
--
Mitch Sundt
Software Engineer
University of Washington
mitche...@gmail.com
--
--
Post: opend...@googlegroups.com
Unsubscribe: opendatakit...@googlegroups.com
Options: http://groups.google.com/group/opendatakit?hl=en
You received this message because you are subscribed to the Google
Groups "ODK Community" group.
To unsubscribe from this group and stop receiving emails from it, send
an email to opendatakit...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.
--
Mitch Sundt
Software Engineer
University of Washington
mitche...@gmail.com

--
Post: opend...@googlegroups.com
Unsubscribe: opendatakit...@googlegroups.com
Options: http://groups.google.com/group/opendatakit?hl=en

You received this message because you are subscribed to a topic in
the Google Groups "ODK Community" group.
To unsubscribe from this topic, visit
https://groups.google.com/d/topic/opendatakit/MBzQgOHKVNc/unsubscribe.
To unsubscribe from this group and all its topics, send an email to
opendatakit...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.
--
Greg Pommen
Biological Sciences Technical Assistant
City of Edmonton Environmental Services
12304-107 St.
Edmonton, AB T5G 2S7
(780) 496-6906 <%28780%29%20496-6906>
fax: (780) 496-4978
greg....@edmonton.ca
--

Post: opendatakit@googlegroups.com
Unsubscribe: opendatakit+unsubscribe@googlegroups.com
Options: http://groups.google.com/group/opendatakit?hl=en

You received this message because you are subscribed to the Google Groups
"ODK Community" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to opendatakit+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
--
Mitch Sundt
Software Engineer
University of Washington
mitchellsundt@gmail.com

--
Post: opendatakit@googlegroups.com
Unsubscribe: opendatakit+unsubscribe@googlegroups.com
Options: http://groups.google.com/group/opendatakit?hl=en

You received this message because you are subscribed to the Google Groups
"ODK Community" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to opendatakit+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--

Post: opendatakit@googlegroups.com
Unsubscribe: opendatakit+unsubscribe@googlegroups.com
Options: http://groups.google.com/group/opendatakit?hl=en

You received this message because you are subscribed to the Google Groups
"ODK Community" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to opendatakit+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--
Mitch Sundt
Software Engineer
University of Washington
mitchellsundt@gmail.com

ODK on Google App Engine keeps running up charges

--

--

--

--

--

--

--

--

--

--

--

--

--

--

--

-- Mitch Sundt Software Engineer University of Washington mitche...@gmail.com

--

-- Mitch Sundt Software Engineer University of Washington mitchellsundt@gmail.com

--

--
Mitch Sundt
Software Engineer
University of Washington
mitche...@gmail.com

--
Mitch Sundt
Software Engineer
University of Washington
mitchellsundt@gmail.com