We have a few thousand records stored on our Aggregate appspot which we
export to a local PC on a weekly basis. Over the past week or two it has
become increasingly problematic to download the data and, since yesterday,
it has not been possible at all. We can generate export data files but,
when we try to save them, we get the following general error message:
"AppEngine Application Problem". The billing settings look ok and we do not
seem to have exceeded the quota. There are no related System Status errors.
However, the frontend error log (attached) shows a
"java.lang.OutOfMemoryError: Java heap space" warning, which I assume is
the problem. I increased the memory size of the server from 128MB to 256MB
but got the same error when trying to export data, and I just upgraded to
Aggregate v1.3.0 but the problem still persists.
I don't really want to delete individual records and I'm not sure what to
do. Any advice would be greatly appreciated.
Thanks,
Glyn
If increasing the JVM size did not resolve the issue, I have no easy fix.
At some point, you will not be able to export the entire contents of your
database in a file download; the root causes are that the entire file needs
to be first read into the server memory then written out over the
connection to the client. This will either eventually exceed the 60-second
request timer limit on Google AppEngine or exceed the virtual memory of the
server. The JVM OutOfMemory error is caused by the later. While the code
could be re-written to stream the data out, you would still have the
60-second request timer limit.
Can you switch to use ODK Briefcase to download data? It is designed
to incrementally
pull only the records that are not already present on your local machine.
Once downloaded, you can then generate a CSV from those files. ODK
Briefcase does not have the column filtering functionality available on the
server and doesn't produce KML files.
Mitch
···
On Wed, Feb 6, 2013 at 12:28 AM, Glyn wrote:
Hi All,
We have a few thousand records stored on our Aggregate appspot which we
export to a local PC on a weekly basis. Over the past week or two it has
become increasingly problematic to download the data and, since yesterday,
it has not been possible at all. We can generate export data files but,
when we try to save them, we get the following general error message:
"AppEngine Application Problem". The billing settings look ok and we do not
seem to have exceeded the quota. There are no related System Status errors.
However, the frontend error log (attached) shows a
"java.lang.OutOfMemoryError: Java heap space" warning, which I assume is
the problem. I increased the memory size of the server from 128MB to
256MB but got the same error when trying to export data, and I just
upgraded to Aggregate v1.3.0 but the problem still persists.
I don't really want to delete individual records and I'm not sure what to
do. Any advice would be greatly appreciated.
Thanks,
Glyn
I was able to pull the data (9997 records) using Briefcase. I noticed that
empty (null) fields on Aggregate appeared as empty cells in the .csv file,
whereas when you export directly from Aggregate, empty fields appear as a
"null" string in the .csv file. This meant that some commands in our Stata
do files, which automatically manage parts of the data, didn't work, Also,
columns were also created for "SumissionDate" and "Key" UUID information,
again not done in the direct export method. Neither of these is a
particular problem, I just wanted to point them out.
Thanks again,
Glyn
···
On Wednesday, February 6, 2013 1:58:17 PM UTC+5:30, Glyn wrote:
>
> Hi All,
>
> We have a few thousand records stored on our Aggregate appspot which we
> export to a local PC on a weekly basis. Over the past week or two it has
> become increasingly problematic to download the data and, since yesterday,
> it has not been possible at all. We can generate export data files but,
> when we try to save them, we get the following general error message:
> "AppEngine Application Problem". The billing settings look ok and we do not
> seem to have exceeded the quota. There are no related System Status errors.
> However, the frontend error log (attached) shows a
> "java.lang.OutOfMemoryError: Java heap space" warning, which I assume is
> the problem. I increased the memory size of the server from 128MB to
> 256MB but got the same error when trying to export data, and I just
> upgraded to Aggregate v1.3.0 but the problem still persists.
>
> I don't really want to delete individual records and I'm not sure what to
> do. Any advice would be greatly appreciated.
> Thanks,
> Glyn
>
The KEY field is needed for linking rows in a repeat-group csv (via
PARENT_KEY in that table) back to their respective parent row in the parent
table; unlike ODK Aggregate, ODK Briefcase will emit a set of csv files,
one for each repeat group.
The submission date should probably be added to the set of metadata that
ODK Aggregate displays (it is present, but not revealed). Submission date
is preserved across downloads and uploads by ODK Briefcase, so you don't
lose track of when the data first entered your systems; in contrast, the
mark-as-complete date reflects the time of last upload prior to being
marked-as-complete (subsequent uploads are ignored and do not alter this
timestamp value).
Mitch
···
On Thu, Feb 7, 2013 at 8:44 PM, Glyn wrote:
Thanks Mitch,
I was able to pull the data (9997 records) using Briefcase. I noticed that
empty (null) fields on Aggregate appeared as empty cells in the .csv file,
whereas when you export directly from Aggregate, empty fields appear as a
"null" string in the .csv file. This meant that some commands in our Stata
do files, which automatically manage parts of the data, didn't work, Also,
columns were also created for "SumissionDate" and "Key" UUID information,
again not done in the direct export method. Neither of these is a
particular problem, I just wanted to point them out.
Thanks again,
Glyn
On Wednesday, February 6, 2013 1:58:17 PM UTC+5:30, Glyn wrote:
Hi All,
We have a few thousand records stored on our Aggregate appspot which we
export to a local PC on a weekly basis. Over the past week or two it has
become increasingly problematic to download the data and, since yesterday,
it has not been possible at all. We can generate export data files but,
when we try to save them, we get the following general error message:
"AppEngine Application Problem". The billing settings look ok and we do not
seem to have exceeded the quota. There are no related System Status errors.
However, the frontend error log (attached) shows a
"java.lang.OutOfMemoryError: Java heap space" warning, which I assume is
the problem. I increased the memory size of the server from 128MB to
256MB but got the same error when trying to export data, and I just
upgraded to Aggregate v1.3.0 but the problem still persists.
I don't really want to delete individual records and I'm not sure what to
do. Any advice would be greatly appreciated.
Thanks,
Glyn