[ODK Community] Aggregate data not appearing

Support
1

You're using a very old version of ODK Aggregate (1.0.6). While I don't
know of any issue that will cause this in 1.0.6, the first step to
resolving this would be to upgrade to ODK Aggregate 1.3.

Be sure to read the release notes for upgrading instructions (i.e.,
publishers will break)
http://code.google.com/p/opendatakit/wiki/AggregateReleaseNotes

On the website submissions tab, I suspect your dataset is beginning to tax
the limits of how we use Google AppEngine. If I change the number of
records viewed on a screen to '10', I can see data when 'none' is selected;
getting the change to 10 to be applied takes a while - the UI keeps wanting
to reset it to 100.

The CSV export should, however, not have this problem. If there are errors
during the CSV export, you would be able to see them by going to the error
logs for the 'background' version of your app.
Getting to the logs page is described here:
http://code.google.com/p/opendatakit/wiki/AppEngineTroubleshooting

If there are any errors being reported, please let me know (or invite me as
a developer).

Mitch

··· On Mon, Feb 11, 2013 at 11:32 AM, wrote:

Hi,
I have an Aggregate instance set up (https://mscantest.appspot.com/) and
created a filter to remove images for export. This has worked fine and
allowed me to see the images for the past year or so, but now if the
submissions are not filtered, I see absolutely nothing, and setting the
filter to no images removes the images that I want to store for reference.

If I export the submissions with no filter, the URL in the csv file that's
exported contains the image, but this is a difficult way to look at images.
Any idea where my fields went?

Thanks,
Kate

--

Post: opendatakit@googlegroups.com
Unsubscribe: opendatakit+unsubscribe@googlegroups.com
Options: http://groups.google.com/group/opendatakit?hl=en

You received this message because you are subscribed to the Google Groups
"ODK Community" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to opendatakit+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

--
Mitch Sundt
Software Engineer
University of Washington
mitchellsundt@gmail.com

2

Hmm. 150 shouldn't be taxing the system at all. I'll take a look...

··· On Mon, Feb 11, 2013 at 1:15 PM, wrote:

On Monday, February 11, 2013 12:33:53 PM UTC-8, Mitch wrote:

You're using a very old version of ODK Aggregate (1.0.6). While I don't
know of any issue that will cause this in 1.0.6, the first step to
resolving this would be to upgrade to ODK Aggregate 1.3.

Be sure
to read the release notes for upgrading instructions (i.e., publishers
will break)
http://code.google.com/p/opendatakit/wiki/AggregateReleaseNotes

On the website submissions tab, I suspect your dataset is beginning to
tax the limits of how we use Google AppEngine. If I change the number of
records viewed on a screen to '10', I can see data when 'none' is selected;
getting the change to 10 to be applied takes a while - the UI keeps wanting
to reset it to 100.

The CSV export should, however, not have this problem. If there are
errors during the CSV export, you would be able to see them by going to the
error logs for the 'background' version of your app.
Getting to the logs page is described here:
http://code.google.com/p/opendatakit/wiki/AppEngineTroubleshooting

If there are any errors being reported, please let me know (or invite me
as a developer).

Mitch

On Mon, Feb 11, 2013 at 11:32 AM, kate.w...@villagereach.org wrote:

Hi,

I have an Aggregate instance set up (https://mscantest.appspot.com/)
and created a filter to remove images for export. This has worked fine and
allowed me to see the images for the past year or so, but now if the
submissions are not filtered, I see absolutely nothing, and setting the
filter to no images removes the images that I want to store for reference.

If I export the submissions with no filter, the URL in the csv file
that's exported contains the image, but this is a difficult way to look at
images. Any idea where my fields went?

Thanks,

Kate

--

--

Post: opend...@googlegroups.com

Unsubscribe: opendatakit...@googlegroups.com

Options: http://groups.google.com/group/opendatakit?hl=en

You received this message because you are subscribed to the Google
Groups "ODK Community" group.

To unsubscribe from this group and stop receiving emails from it, send
an email to opendatakit...@googlegroups.com.

For more options, visit https://groups.google.com/groups/opt_out.

--
Mitch Sundt
Software Engineer
University of Washington
mitche...@gmail.com

I've upgraded to Aggregate 1.3.0 but am still having troubles - I see that
changing the number of submissions per page helps, but the images are now
very blurry. I've added you as a developer, and ideally I'd like to find a
way to be able to store the whole data set (maybe 150 submissions, all with
images) - is there a better method than Aggregate to do this?

Thanks.
Kate

--

Post: opendatakit@googlegroups.com
Unsubscribe: opendatakit+unsubscribe@googlegroups.com
Options: http://groups.google.com/group/opendatakit?hl=en

You received this message because you are subscribed to the Google Groups
"ODK Community" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to opendatakit+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

--
Mitch Sundt
Software Engineer
University of Washington
mitchellsundt@gmail.com

3

I've opened this ticket against Google:
http://code.google.com/p/googleappengine/issues/detail?id=8850 But don't
really expect to hear anything back on it. I suspect the reported error is
referring to the overall 60-seconds HTTP request limit, and not a datastore
limit.

Google AppEngine has a hard 60-second limit to all website requests, and
this is apparently hitting that limit. With 300 images in the Scan output,
a 100-record query would generate a lot of datastore requests:

1 + 100*300 = 30,001 (1 request to get the 100 records, plus 1 request for
information about each image of each row)

The latency for datastore requests is close to 15 milliseconds (
http://code.google.com/status/appengine/detail/datastore/2013/02/19#ae-trust-detail-datastore-get-latency).

So the default viewing of 100 records will definitely trip that 60-second
limit:

30,001 * 15ms = 450,045 ms = 7+ minutes.

Which is why if you reduce the number of rows to view to 10 per page, you
can see the data with the images:

10*300 = 3,000 * 15ms = 45,000ms -> 45 seconds.

The upshot is that an ODK Scan survey of this size (~ 300 text fields) is
better hosted on a Tomcat server where you can configure a longer request
timeout.

I've opened this Aggregate issue:
http://code.google.com/p/opendatakit/issues/detail?id=768 -- the
rows-per-page value should not be reset. Once you set it, it should be
fixed and continue to be honored, but it is being reset to 100, which makes
using Google AppEngine that much more impossible in this instance.

Mitch

··· On Wed, Feb 13, 2013 at 6:26 AM, wrote:

Mitch, any idea what's going on?

--

Post: opendatakit@googlegroups.com
Unsubscribe: opendatakit+unsubscribe@googlegroups.com
Options: http://groups.google.com/group/opendatakit?hl=en

You received this message because you are subscribed to the Google Groups
"ODK Community" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to opendatakit+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

--
Mitch Sundt
Software Engineer
University of Washington
mitchellsundt@gmail.com