Has anyone ever used the ODK platform to collect more than 10000 forms? I have been using ODK collect + Aggregate (in Google App Engine) and after something like 8000 submissions the ODK aggregate application deployed in Google App Engine can't export the .csv files. I'm still able to download the data through ODK Briefcase (now fetching 1465 outo of ?), but I'm worried if the platform is stable and robust enough to support something around 30000 submissions. Have anyone used it for something this size?
My form has the collection of around 30 string variables, 5 decimal, 3 images, and coordinates.
You are running out of memory on AppEngine -- in order to generate the CSV,
it must generate the entire thing in memory. To increase memory, you need
to increase the JVM size, which, for AppEngine, means you need to increase
the "Frontend Instance Class" -- go to your AppEngine dashboard, click on
"Application Settings" and change this setting. Because of the in-memory
requirement, for larger datasets, we recommend using ODK Briefcase to pull
the data down to your computer, and generate the CSV there, where we can
stream the data directly into a file and not need to hold anything in
memory.
Others have used the Form Management / Purge Submissions button to remove
older data from ODK Aggregate after they have pulled the data down to ODK
Briefcase. That keeps the data storage usage on ODK Aggregate low, and
makes your computer the holder of all of the data-of-record for your
survey, so you would want to make sure you keep backups of your briefcase
storage location.
···
On Wed, Dec 17, 2014 at 12:37 AM, wrote:
>
> Has anyone ever used the ODK platform to collect more than 10000 forms? I
> have been using ODK collect + Aggregate (in Google App Engine) and after
> something like 8000 submissions the ODK aggregate application deployed in
> Google App Engine can't export the .csv files. I'm still able to download
> the data through ODK Briefcase (now fetching 1465 outo of ?), but I'm
> worried if the platform is stable and robust enough to support something
> around 30000 submissions. Have anyone used it for something this size?
> My form has the collection of around 30 string variables, 5 decimal, 3
> images, and coordinates.
>
> Thanks,
>
> Davi
>
> --
> --
> Post: opendatakit@googlegroups.com
> Unsubscribe: opendatakit+unsubscribe@googlegroups.com
> Options: http://groups.google.com/group/opendatakit?hl=en
>
> ---
> You received this message because you are subscribed to the Google Groups
> "ODK Community" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to opendatakit+unsubscribe@googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.
>
Thanks for the help, Mitch. I'm downloading the submissions to my PC using
ODK Briefcase (10920 so far). As soon as that completes and I have a
external backup, I'll use the suggested action of purging the existing
submissions.
···
Em quarta-feira, 17 de dezembro de 2014 16h35min30s UTC-2, Mitch Sundt escreveu:
>
> You are running out of memory on AppEngine -- in order to generate the
> CSV, it must generate the entire thing in memory. To increase memory, you
> need to increase the JVM size, which, for AppEngine, means you need to
> increase the "Frontend Instance Class" -- go to your AppEngine dashboard,
> click on "Application Settings" and change this setting. Because of the
> in-memory requirement, for larger datasets, we recommend using ODK
> Briefcase to pull the data down to your computer, and generate the CSV
> there, where we can stream the data directly into a file and not need to
> hold anything in memory.
>
> Others have used the Form Management / Purge Submissions button to remove
> older data from ODK Aggregate after they have pulled the data down to ODK
> Briefcase. That keeps the data storage usage on ODK Aggregate low, and
> makes your computer the holder of all of the data-of-record for your
> survey, so you would want to make sure you keep backups of your briefcase
> storage location.
>
>
>
>
>
> On Wed, Dec 17, 2014 at 12:37 AM, <davisch...@gmail.com > wrote:
>>
>> Has anyone ever used the ODK platform to collect more than 10000 forms? I
>> have been using ODK collect + Aggregate (in Google App Engine) and after
>> something like 8000 submissions the ODK aggregate application deployed in
>> Google App Engine can't export the .csv files. I'm still able to download
>> the data through ODK Briefcase (now fetching 1465 outo of ?), but I'm
>> worried if the platform is stable and robust enough to support something
>> around 30000 submissions. Have anyone used it for something this size?
>> My form has the collection of around 30 string variables, 5 decimal, 3
>> images, and coordinates.
>>
>> Thanks,
>>
>> Davi
>>
>> --
>> --
>> Post: opend...@googlegroups.com
>> Unsubscribe: opendatakit...@googlegroups.com
>> Options: http://groups.google.com/group/opendatakit?hl=en
>>
>> ---
>> You received this message because you are subscribed to the Google Groups
>> "ODK Community" group.
>> To unsubscribe from this group and stop receiving emails from it, send an
>> email to opendatakit...@googlegroups.com .
>> For more options, visit https://groups.google.com/d/optout.
>>
>
>
> --
> Mitch Sundt
> Software Engineer
> University of Washington
> mitche...@gmail.com
>