ODK Aggregate to Google Fusion Tables Link Stopping?

Hi.

I've successfully produced a small survey form which captures just
four fields - location, date-time, photo & notes.
The data uploads OK to ODK Aggregate and I've established a connection
to Google Fusion Tables using the "BOTH Upload Existing and Stream New
Submission Data" option.

This worked correctly for about 10 days during which time we uploaded
about 30 forms - all of which appeared both in ODK Aggregate and in
the Google Fusion table and map views.

** At this point I downloaded some of the data from the Google Fusion
Table using placemat python code. **

However we uploaded a further 10 forms yesterday - these appear
corectly in ODK Aggregate - but haven't appeared in Google Fusion.

I tried re-connecting to Google Fusion from within ODK Aggregate - but
this results in new tables being generated.

I've overcome the problem in the short term by deleting the Google
Fusion tables and starting again. But I'm interested to know whether
there is anything I can do to - either to overcome this problem - or
to help debug or otherwise characterise the problem.

Many Thanks, Kirby

I note a post on the Google Fusion Tables Group

http://groups.google.com/group/fusion-tables-users-group/browse_thread/thread/5f2d3a1e971c92a8/142ea1f71c61562d?lnk=gst&q=odk#142ea1f71c61562d

which says:-

··· ---------- Forwarded message ---------- From: W. Brunette - Dec 2010

When a submission is posted to aggregate it also posts the submission
to fusion table (when streaming). With the app engine 30 second time
out there may be an issue if lots of data is being transmitted. There
could be other reasons like if they expected they have all the data
uploaded to aggregate first it will only move the data up until the 30
second time out then it will stop. This will be fixed in
Aggregate-1.x, the alpha of which is expected out next week.

Could this explain the issue? - I'm using Aggregate 0.9.4

The data that was added contained 10 forms each containing a 1.4 MByte
photo. Previously we had loaded over 20 such forms.

Thanks Kirby

I've now had a look at the 'Google App Engine Dashboard' and it shows
that I've reached 100% of my "Outgoing Bandwidth" quota.
The "Total Stored Data" is at 9% of its quota and all other values are
at 1% or less than their quota.

Would this explain the inability of ODK Aggregate to update the Google
Fusion Tables?

Incidentaly I can still view the data in table and map format - but
the photo images are not displayed - I assume that this is as a result
of the "Outgoing Bandwidth" quota being reached.

The Submission logs contain errors/warnings for each form uploaded:-

  1. 2011-04-08 06:49:52.435 /submission 201 7693ms 6914cpu_ms
    1057api_cpu_ms 0kb gzip(gfe) See details

86.185.138.96 - - [08/Apr/2011:06:49:52 -0700] "POST /submission HTTP/
1.1" 201 79 - "gzip(gfe)" "my-app.appspot.com" ms=7694 cpu_ms=6914
api_cpu_ms=1058 cpm_usd=0.323304 loading_request=1

  1. W 2011-04-08 06:49:48.193

org.datanucleus.store.appengine.MetaDataValidator warn: Meta-data
warning for org.odk.aggregate.form.Form.elementTreeRoot: The datastore
does not support joins and therefore cannot honor requests to eagerly
load related objects. The field will be fetched lazily on first
access. You can modify this warning by setting the
datanucleus.appengine.ignorableMetaDataBehavior property in your
config. A value of NONE will silence the warning. A value of ERROR
will turn the warning into an exception.

  1. I 2011-04-08 06:49:52.434

This request caused a new process to be started for your application,
and thus caused your application code to be loaded for the first time.
This request may thus take longer and use more CPU than a typical
request for your application.

Is this normal?

Many Thanks Kirby

Are you running Aggregate 0.9.5? There have been a number of changes in
Google's infrastructure that have broken the publication of data from
Aggregate 0.9.4 and earlier into Google Spreadsheets and Fusion Tables.

See:
http://groups.google.com/group/opendatakit/browse_thread/thread/6afa0cf6b932c280#

Mitch

··· On Fri, Apr 8, 2011 at 7:33 AM, Cycle Exeter wrote:

I've now had a look at the 'Google App Engine Dashboard' and it shows
that I've reached 100% of my "Outgoing Bandwidth" quota.
The "Total Stored Data" is at 9% of its quota and all other values are
at 1% or less than their quota.

Would this explain the inability of ODK Aggregate to update the Google
Fusion Tables?

Incidentaly I can still view the data in table and map format - but
the photo images are not displayed - I assume that this is as a result
of the "Outgoing Bandwidth" quota being reached.

The Submission logs contain errors/warnings for each form uploaded:-

  1. 2011-04-08 06:49:52.435 /submission 201 7693ms 6914cpu_ms
    1057api_cpu_ms 0kb gzip(gfe) See details

86.185.138.96 - - [08/Apr/2011:06:49:52 -0700] "POST /submission HTTP/
1.1" 201 79 - "gzip(gfe)" "my-app.appspot.com" ms=7694 cpu_ms=6914
api_cpu_ms=1058 cpm_usd=0.323304 loading_request=1

  1. W 2011-04-08 06:49:48.193

org.datanucleus.store.appengine.MetaDataValidator warn: Meta-data
warning for org.odk.aggregate.form.Form.elementTreeRoot: The datastore
does not support joins and therefore cannot honor requests to eagerly
load related objects. The field will be fetched lazily on first
access. You can modify this warning by setting the
datanucleus.appengine.ignorableMetaDataBehavior property in your
config. A value of NONE will silence the warning. A value of ERROR
will turn the warning into an exception.

  1. I 2011-04-08 06:49:52.434

This request caused a new process to be started for your application,
and thus caused your application code to be loaded for the first time.
This request may thus take longer and use more CPU than a typical
request for your application.

Is this normal?

Many Thanks Kirby

--
Post: opendatakit@googlegroups.com
Unsubscribe: opendatakit+unsubscribe@googlegroups.com
Options: http://groups.google.com/group/opendatakit?hl=en

--
Mitch Sundt
Software Engineer


University of Washington
mitchellsundt@gmail.com

Yes it is normal to receive the warning about not supporting joins

Google will terminate your app engine process to save CPU cycles when
your application has not been used in a while. The first call to any
Aggregate functionality can take quite a while as Google reloads the
application. Once it's reloaded the subsequent calls are quite fast.
This long process of reload has caused so many errors that Google now
logs so that you know when they are reloading because the behavior
will not be quite right.

I have not seen running out of outgoing external bandwidth quota
before as most of the time you run out of the other resources first.
Can you give us more details? Did you run briefcase that day or
something?

Waylon

··· On Fri, Apr 8, 2011 at 10:20 AM, Mitch Sundt wrote: > Are you running Aggregate 0.9.5? There have been a number of changes in > Google's infrastructure that have broken the publication of data from > Aggregate 0.9.4 and earlier into Google Spreadsheets and Fusion Tables. > > See: > http://groups.google.com/group/opendatakit/browse_thread/thread/6afa0cf6b932c280# > > Mitch > > On Fri, Apr 8, 2011 at 7:33 AM, Cycle Exeter wrote: >> >> I've now had a look at the 'Google App Engine Dashboard' and it shows >> that I've reached 100% of my "Outgoing Bandwidth" quota. >> The "Total Stored Data" is at 9% of its quota and all other values are >> at 1% or less than their quota. >> >> Would this explain the inability of ODK Aggregate to update the Google >> Fusion Tables? >> >> Incidentaly I can still view the data in table and map format - but >> the photo images are not displayed - I assume that this is as a result >> of the "Outgoing Bandwidth" quota being reached. >> >> The Submission logs contain errors/warnings for each form uploaded:- >> >> 1. 2011-04-08 06:49:52.435 /submission 201 7693ms 6914cpu_ms >> 1057api_cpu_ms 0kb gzip(gfe) See details >> >> 86.185.138.96 - - [08/Apr/2011:06:49:52 -0700] "POST /submission HTTP/ >> 1.1" 201 79 - "gzip(gfe)" "my-app.appspot.com" ms=7694 cpu_ms=6914 >> api_cpu_ms=1058 cpm_usd=0.323304 loading_request=1 >> >> 2. W 2011-04-08 06:49:48.193 >> >> org.datanucleus.store.appengine.MetaDataValidator warn: Meta-data >> warning for org.odk.aggregate.form.Form.elementTreeRoot: The datastore >> does not support joins and therefore cannot honor requests to eagerly >> load related objects. The field will be fetched lazily on first >> access. You can modify this warning by setting the >> datanucleus.appengine.ignorableMetaDataBehavior property in your >> config. A value of NONE will silence the warning. A value of ERROR >> will turn the warning into an exception. >> >> 3. I 2011-04-08 06:49:52.434 >> >> This request caused a new process to be started for your application, >> and thus caused your application code to be loaded for the first time. >> This request may thus take longer and use more CPU than a typical >> request for your application. >> >> Is this normal? >> >> Many Thanks Kirby >> >> -- >> Post: opendatakit@googlegroups.com >> Unsubscribe: opendatakit+unsubscribe@googlegroups.com >> Options: http://groups.google.com/group/opendatakit?hl=en > > > > -- > Mitch Sundt > Software Engineer > http://www.OpenDataKit.org > University of Washington > mitchellsundt@gmail.com > > -- > Post: opendatakit@googlegroups.com > Unsubscribe: opendatakit+unsubscribe@googlegroups.com > Options: http://groups.google.com/group/opendatakit?hl=en >

Mich,

Yes I'm running Aggregate 0.9.5. If in the future I upgrade to 1.x.x

  • will it be possible to transfer the data I've already captured?

I've addded some more forms - which appear in Aggregate - but still
(12 hours later) haven't appeared in the Fussion Tables. If they fail
in this way - will they ever catch up?

Is this failure related to exceeding some quota within Fussion Tables?

Does ODK Aggregate (or it's infrastructure) have capacity limits I
should be aware of? I've currently uploaed about 40 forms each of
which contain a 1.5 MByte image (i.e. 60 MBytes total)

Many Thanks

Kirby

Hi Kirby,

Waylon can help diagnose the publication problems; he's our expert in this
area.

w.r.t. migration. We are still figuring out what support we will have for
uploading datasets into Aggregate 1.0.

w.r.t. capacity limits. I am aware of no limits in 0.9.x or 1.0.

Mitch

··· On Sat, Apr 9, 2011 at 1:05 AM, Cycle Exeter wrote:

Mich,

Yes I'm running Aggregate 0.9.5. If in the future I upgrade to 1.x.x

  • will it be possible to transfer the data I've already captured?

I've addded some more forms - which appear in Aggregate - but still
(12 hours later) haven't appeared in the Fussion Tables. If they fail
in this way - will they ever catch up?

Is this failure related to exceeding some quota within Fussion Tables?

Does ODK Aggregate (or it's infrastructure) have capacity limits I
should be aware of? I've currently uploaed about 40 forms each of
which contain a 1.5 MByte image (i.e. 60 MBytes total)

Many Thanks

Kirby

--
Post: opendatakit@googlegroups.com
Unsubscribe: opendatakit+unsubscribe@googlegroups.com
Options: http://groups.google.com/group/opendatakit?hl=en

--
Mitch Sundt
Software Engineer


University of Washington
mitchellsundt@gmail.com

Actually, Aggregate has capacity limits. When running on Google App
Engine in the cloud you are subject to their limits. Here is a link to
their quota page

http://code.google.com/appengine/docs/quotas.html

You can check you quota usage on Google App Engine console by going to
appengine.appspot.com

The aggregate 0.9.x series will not "catch up" because the data
sending is done in the foreground. In the unreleased Aggregate 1.x the
work moves to the background and Aggregate will retry. Because of
Google propagation delays and such it can take minutes for you to see
something; however, past an hour I don't think the system will make
something appear.

To be clear, you have to create a fusion table connection for each
form. Aggregate does not automatically create a fusion table
connection when you upload the form. You have to explicitly create the
connection for each form.

Waylon

··· On Mon, Apr 11, 2011 at 9:48 AM, Mitch Sundt wrote: > Hi Kirby, > > Waylon can help diagnose the publication problems; he's our expert in this > area. > > w.r.t. migration. We are still figuring out what support we will have for > uploading datasets into Aggregate 1.0. > > w.r.t. capacity limits. I am aware of no limits in 0.9.x or 1.0. > > Mitch > > On Sat, Apr 9, 2011 at 1:05 AM, Cycle Exeter wrote: >> >> Mich, >> >> Yes I'm running Aggregate 0.9.5. If in the future I upgrade to 1.x.x >> - will it be possible to transfer the data I've already captured? >> >> I've addded some more forms - which appear in Aggregate - but still >> (12 hours later) haven't appeared in the Fussion Tables. If they fail >> in this way - will they ever catch up? >> >> Is this failure related to exceeding some quota within Fussion Tables? >> >> Does ODK Aggregate (or it's infrastructure) have capacity limits I >> should be aware of? I've currently uploaed about 40 forms each of >> which contain a 1.5 MByte image (i.e. 60 MBytes total) >> >> Many Thanks >> >> >> Kirby >> >> -- >> Post: opendatakit@googlegroups.com >> Unsubscribe: opendatakit+unsubscribe@googlegroups.com >> Options: http://groups.google.com/group/opendatakit?hl=en > > > > -- > Mitch Sundt > Software Engineer > http://www.OpenDataKit.org > University of Washington > mitchellsundt@gmail.com > > -- > Post: opendatakit@googlegroups.com > Unsubscribe: opendatakit+unsubscribe@googlegroups.com > Options: http://groups.google.com/group/opendatakit?hl=en >

Hi Waylon,

Thanks for the reply.

To be clear, you have to create a fusion table connection for each
form.

Yes - I've created a Fusion table for my form.

At first when I uploaded data (using ODK Connect) it would appear in
Aggregate and then (within minutes) in the corresponding Fusion Table
and Map. After a while (approx 25 entries) form data continued to
arrive correctly in Aggregate - but was no longer being added to the
Fusion Table (I had chosen the "BOTH Upload Existing and Stream New
Submission Data" option).

The only way I could get more data into Fusion was by deleting the
Fusion table completely and then re-establishing the "External Service
Connection".

Since the initial failure this has happened twice more - and in both
cases I had to delete and re-create the Fusion Table. Since the
initial failure none of the subsequently submited form data has made
it through to Fusion!

Any suggestions very welcome!!

Kirby

Actually, Aggregate has capacity limits. When running on Google App
Engine in the cloud you are subject to their limits. Here is a link to
their quota page

http://code.google.com/appengine/docs/quotas.html

You can check you quota usage on Google App Engine console by going to
appengine.appspot.com

The aggregate 0.9.x series will not "catch up" because the data
sending is done in the foreground. In the unreleased Aggregate 1.x the
work moves to the background and Aggregate will retry. Because of
Google propagation delays and such it can take minutes for you to see
something; however, past an hour I don't think the system will make
something appear.

Aggregate does not automatically create a fusion table

··· On Apr 11, 7:38 pm, "W. Brunette" wrote: > connection when you upload the form. You have to explicitly create the > connection for each form. > > Waylon > > On Mon, Apr 11, 2011 at 9:48 AM, Mitch Sundt wrote: > > Hi Kirby, > > > Waylon can help diagnose the publication problems; he's our expert in this > > area. > > > w.r.t. migration. We are still figuring out what support we will have for > > uploading datasets into Aggregate 1.0. > > > w.r.t. capacity limits. I am aware of no limits in 0.9.x or 1.0. > > > Mitch > > > On Sat, Apr 9, 2011 at 1:05 AM, Cycle Exeter wrote: > > >> Mich, > > >> Yes I'm running Aggregate 0.9.5. If in the future I upgrade to 1.x.x > >> - will it be possible to transfer the data I've already captured? > > >> I've addded some more forms - which appear in Aggregate - but still > >> (12 hours later) haven't appeared in the Fussion Tables. If they fail > >> in this way - will they ever catch up? > > >> Is this failure related to exceeding some quota within Fussion Tables? > > >> Does ODK Aggregate (or it's infrastructure) have capacity limits I > >> should be aware of? I've currently uploaed about 40 forms each of > >> which contain a 1.5 MByte image (i.e. 60 MBytes total) > > >> Many Thanks > > >> Kirby > > >> -- > >> Post: opendatakit@googlegroups.com > >> Unsubscribe: opendatakit+unsubscribe@googlegroups.com > >> Options:http://groups.google.com/group/opendatakit?hl=en > > > -- > > Mitch Sundt > > Software Engineer > >http://www.OpenDataKit.org > > University of Washington > > mitchellsu...@gmail.com > > > -- > > Post: opendatakit@googlegroups.com > > Unsubscribe: opendatakit+unsubscribe@googlegroups.com > > Options:http://groups.google.com/group/opendatakit?hl=en > >