We've been successfully using ODK Collect 1.1.7 RC2 with Aggregate 1.7 RC2
for a large household health survey. We check and download submitted data to
a local server daily but leave the raw data on Aggregate for data safety. We
have not yet been using Google Fusion. We've noticed that viewing data on
Aggregate has become increasingly slow (there are 300+ fields in our form)
to the point that, even with a fast internet connection, the submissions
page freezes without displaying any records. We have now run into the
problem that once 1000 records were uploaded, new data from the phones no
longer appear. Searching the community brought up at least one other user
(Ryan) with the same issue, but I couldn't really see the fix. Because we
still had the un-uploaded data on the phones, I deleted the form (and,
therefore, the data) from Aggregate and reloaded it, after which we could
continue submitting data as before. I looked at our App Engine - there were
timeout errors but I could no see if they were specifically related to this
problem
Could someone please explain why this issue occurs (seemingly when 1000
records have been uploaded) and how to overcome it (preferably without
deleting the entire form and data each time)? Please bear in mind that
although I am competent in form design, XML coding and general management of
ODK, I have no technical or development knowledge.
Any help would be greatly appreciated. Many thanks in advance.
Glyn
I'm glad you are successfully using Aggregate. To clarify, I believe you
are using Aggregate 0.9.x.
If you go to the home page of your server, you'll likely see a "v0.9.x" text
to the far right of a top row of links. Your server home page should also
be very different from that shown at http://opendatakit.appspot.com (the new
ODK Aggregate 1.0 look).
Aggregate 0.9.x has a 1000-record limit to the number of records it can
display (this is by design); it does, however, still retain all records --
it will hold as many records as you've uploaded and does not loose or ignore
the 1001st record, etc.; in general, any webserver will have a limit to what
it can display interactively (at some point it will run out of memory or
time out).
On Aggregate 0.9.x, we provide the Briefcase applet (one of the links on the
top row) to allow you to extract your data from the server into a local CSV
file for local processing. That should retrieve all data within the server,
and should work regardless of the number of submissions you have uploaded.
If you are running an Aggregate version prior to 0.9.8, please consider
upgrading to 0.9.8; that version is backward-compatible to all earlier 0.9.x
releases. It has new support for downloading submissions as submission XML
using the new Briefcase application (downloadable from the here: http://code.google.com/p/opendatakit/downloads/list ). That application is
the mechanism by which you can migrate your dataset to Aggregate 1.0 and
move off Aggregate 0.9.x (by downloading the data to your local machine then
uploading it into the new Aggregate instance; this functionality works right
now, so you might want to explore moving to Aggregate 1.0). The Briefcase
application does not yet, but will eventually, have the same csv-generation
capability as the Briefcase applet it is replacing. If you have repeating
groups within your survey, without this feature, you will have less
functionality to examine your data in Aggregate 1.0 than in Aggregate 0.9.x,
so you may want to keep your production server as a 0.9.x server until we
finish the Briefcase application and make it on par with the applet
available on Aggregate 0.9.x.
Going forward, Aggregate 1.0 has far fewer display restrictions; the latest
iteration, RC2, has implemented display paging on the submissions display
page so you can page through all the submissions on the server, rather than
just the first 1000 (and you get to set the page size, as well).
Mitch
···
On Tue, Oct 11, 2011 at 4:35 AM, wrote:
Hi all,
We've been successfully using ODK Collect 1.1.7 RC2 with Aggregate 1.7 RC2
for a large household health survey. We check and download submitted data to
a local server daily but leave the raw data on Aggregate for data safety. We
have not yet been using Google Fusion. We've noticed that viewing data on
Aggregate has become increasingly slow (there are 300+ fields in our form)
to the point that, even with a fast internet connection, the submissions
page freezes without displaying any records. We have now run into the
problem that once 1000 records were uploaded, new data from the phones no
longer appear. Searching the community brought up at least one other user
(Ryan) with the same issue, but I couldn't really see the fix. Because we
still had the un-uploaded data on the phones, I deleted the form (and,
therefore, the data) from Aggregate and reloaded it, after which we could
continue submitting data as before. I looked at our App Engine - there were
timeout errors but I could no see if they were specifically related to this
problem
Could someone please explain why this issue occurs (seemingly when 1000
records have been uploaded) and how to overcome it (preferably without
deleting the entire form and data each time)? Please bear in mind that
although I am competent in form design, XML coding and general management of
ODK, I have no technical or development knowledge.
Any help would be greatly appreciated. Many thanks in advance.
Glyn
I've added you as site admin and sent you an e-mail with our appspot
url.
Many thanks,
Glyn
···
On Oct 12, 5:54 pm, Mitch Sundt wrote:
> If you're on appspot, can you add me as a user (mitchellsu...@gmail.com)
> with site admin privileges with ODK Aggregate?
> And send me your appspot link.
>
> Mitch
>
> On Wed, Oct 12, 2011 at 2:18 AM, wrote:
> > Hi Mitch,
>
> > I apologize - we are already running Aggregate 1.0 RC2 (not 1.7, as I'd
> > written - we do see the data displayed as in
> >http://opendatakit.appspot.com/Aggregate.html#submissions/filter///), so
> > I'm unclear as to why we appear to have the 1000 records restriction.
>
> > Any further advice?
> > Thanks, and sorry for the mistake.
> > Glyn
>
> > --
> > Post: opendatakit@googlegroups.com
> > Unsubscribe: opendatakit+unsubscribe@googlegroups.com
> > Options:http://groups.google.com/group/opendatakit?hl=en
>
> --
> Mitch Sundt
> Software Engineerhttp://www.OpenDataKit.org
> University of Washington
> mitchellsu...@gmail.com