Overload of data on Server

Hi Guys,

We have over a thousand xml data that has been uploaded onto the ODK
Aggregate Server. When we try to export to get a .csv file, we get
the following error:

Failure - abandoned all retry attempts.

Is this because of an excess of data? Is there a capacity issue with
the server? We have enabled billing for it.

I tried using the Kobo Offline Processor but the sync doesn't word so
I stored the xml data on my computer and used transcribe. It created
a csv file but all the variables weren't shown. It only showed the
Section names along with combined data.

Does anyone know of another way to get the csv file? VBA maybe?

Thanks in advance,

Sanjeev

what version of the server are you using?

also, have you tried briefcase? it should be able to do much of what
kobo offline processor does.

··· On Wed, Nov 23, 2011 at 08:46, Sanjeev Swamy wrote: > Hi Guys, > > We have over a thousand xml data that has been uploaded onto the ODK > Aggregate Server. When we try to export to get a .csv file, we get > the following error: > > Failure - abandoned all retry attempts. > > Is this because of an excess of data? Is there a capacity issue with > the server? We have enabled billing for it. > > I tried using the Kobo Offline Processor but the sync doesn't word so > I stored the xml data on my computer and used transcribe. It created > a csv file but all the variables weren't shown. It only showed the > Section names along with combined data. > > Does anyone know of another way to get the csv file? VBA maybe? > > Thanks in advance, > > Sanjeev > > -- > Post: opendatakit@googlegroups.com > Unsubscribe: opendatakit+unsubscribe@googlegroups.com > Options: http://groups.google.com/group/opendatakit?hl=en >

Hi Yaw,

I am using ODK Aggregate v 1.0.1

When I tried using ODK Briefcase, I get the following error while
retrieving data:

NOT ALL SUBMISSIONS RETRIEVED: Error fetching list of submissions:
Fetch of submission download chunk failed. Detailed error: Unexpected
java.net.SocketTimeoutException: Read timed out
FAILED.

Thanks,

Sanjeev

··· On Nov 23, 12:00 pm, Yaw Anokwa wrote: > what version of the server are you using? > > also, have you tried briefcase? it should be able to do much of what > kobo offline processor does. > > > > > > > > On Wed, Nov 23, 2011 at 08:46, Sanjeev Swamy wrote: > > Hi Guys, > > > We have over a thousand xml data that has been uploaded onto the ODK > > Aggregate Server. When we try to export to get a .csv file, we get > > the following error: > > > Failure - abandoned all retry attempts. > > > Is this because of an excess of data? Is there a capacity issue with > > the server? We have enabled billing for it. > > > I tried using the Kobo Offline Processor but the sync doesn't word so > > I stored the xml data on my computer and used transcribe. It created > > a csv file but all the variables weren't shown. It only showed the > > Section names along with combined data. > > > Does anyone know of another way to get the csv file? VBA maybe? > > > Thanks in advance, > > > Sanjeev > > > -- > > Post: opendatakit@googlegroups.com > > Unsubscribe: opendatakit+unsubscribe@googlegroups.com > > Options:http://groups.google.com/group/opendatakit?hl=en

I assume this is an AppEngine instance...
(1) Please go to the AppEngine dashboard (via
https://appengine.google.com/), select the "Permissions" link on the
left sidebar and invite me (
mitchellsundt@gmail.com ) as a Developer to your AppEngine instance.
(2) Within ODK Aggregate, please add me with Site Admin privileges.

And let me know what your appengine URL is.

Thanks,

Mitch

··· On Wed, Nov 23, 2011 at 9:40 AM, Sanjeev Swamy wrote:

Hi Yaw,

I am using ODK Aggregate v 1.0.1

When I tried using ODK Briefcase, I get the following error while
retrieving data:

NOT ALL SUBMISSIONS RETRIEVED: Error fetching list of submissions:
Fetch of submission download chunk failed. Detailed error: Unexpected
java.net.SocketTimeoutException: Read timed out
FAILED.

Thanks,

Sanjeev

On Nov 23, 12:00 pm, Yaw Anokwa yano...@gmail.com wrote:

what version of the server are you using?

also, have you tried briefcase? it should be able to do much of what
kobo offline processor does.

On Wed, Nov 23, 2011 at 08:46, Sanjeev Swamy ssw...@poverty-action.org wrote:

Hi Guys,

We have over a thousand xml data that has been uploaded onto the ODK
Aggregate Server. When we try to export to get a .csv file, we get
the following error:

Failure - abandoned all retry attempts.

Is this because of an excess of data? Is there a capacity issue with
the server? We have enabled billing for it.

I tried using the Kobo Offline Processor but the sync doesn't word so
I stored the xml data on my computer and used transcribe. It created
a csv file but all the variables weren't shown. It only showed the
Section names along with combined data.

Does anyone know of another way to get the csv file? VBA maybe?

Thanks in advance,

Sanjeev

--
Post: opendatakit@googlegroups.com
Unsubscribe: opendatakit+unsubscribe@googlegroups.com
Options:http://groups.google.com/group/opendatakit?hl=en

--
Post: opendatakit@googlegroups.com
Unsubscribe: opendatakit+unsubscribe@googlegroups.com
Options: http://groups.google.com/group/opendatakit?hl=en

--
Mitch Sundt
Software Engineer
University of Washington
mitchellsundt@gmail.com

Hi..

I'm also getting the same error, I'm using google AppEngine instances,
below is the error log

2011-11-28 02:45:59.872

Error for /gae/csvGeneratorTask
java.lang.OutOfMemoryError: Java heap space
at java.nio.ByteBuffer.wrap(ByteBuffer.java:367)
at java.lang.StringCoding$StringDecoder.decode(StringCoding.java:155)
at java.lang.StringCoding.decode(StringCoding.java:191)
at java.lang.String.(String.java:451)
at
com.google.appengine.repackaged.com.google.io.protocol.ProtocolSupport.toStringUtf8(ProtocolSupport.java:
516)
at
com.google.appengine.repackaged.com.google.io.protocol.ProtocolSupport.toStringUtf8(ProtocolSupport.java:
503)
at com.google.storage.onestore.v3.OnestoreEntity
$Property.getName(OnestoreEntity.java:2989)
at
com.google.appengine.api.datastore.DataTypeTranslator.addPropertyValueToMap(DataTypeTranslator.java:
309)
at
com.google.appengine.api.datastore.DataTypeTranslator.extractIndexedPropertiesFromPb(DataTypeTranslator.java:
217)
at
com.google.appengine.api.datastore.DataTypeTranslator.extractPropertiesFromPb(DataTypeTranslator.java:
238)
at
com.google.appengine.api.datastore.EntityTranslator.createFromPb(EntityTranslator.java:
29)
at
com.google.appengine.api.datastore.QueryResultsSourceImpl.processQueryResult(QueryResultsSourceImpl.java:
153)
at
com.google.appengine.api.datastore.QueryResultsSourceImpl.loadMoreEntities(QueryResultsSourceImpl.java:
86)
at
com.google.appengine.api.datastore.QueryResultsSourceImpl.loadMoreEntities(QueryResultsSourceImpl.java:
71)
at
com.google.appengine.api.datastore.QueryResultIteratorImpl.ensureLoaded(QueryResultIteratorImpl.java:
156)
at
com.google.appengine.api.datastore.QueryResultIteratorImpl.hasNext(QueryResultIteratorImpl.java:
65)
at
org.opendatakit.common.persistence.engine.gae.QueryImpl.fetchResults(QueryImpl.java:
844)
at
org.opendatakit.common.persistence.engine.gae.QueryImpl.chunkFetch(QueryImpl.java:
776)
at
org.opendatakit.common.persistence.engine.gae.QueryImpl.coreExecuteQuery(QueryImpl.java:
1035)
at
org.opendatakit.common.persistence.engine.gae.QueryImpl.executeQuery(QueryImpl.java:
1141)
at
org.opendatakit.aggregate.submission.type.ChoiceSubmissionType.getValueFromEntity(ChoiceSubmissionType.java:
80)
at
org.opendatakit.aggregate.submission.SubmissionSet.buildSubmissionFields(SubmissionSet.java:
481)
at
org.opendatakit.aggregate.submission.SubmissionSet.(SubmissionSet.java:
283)
at
org.opendatakit.aggregate.submission.Submission.(Submission.java:
75)
at
org.opendatakit.aggregate.query.submission.QueryByUIFilterGroup.getResultSubmissions(QueryByUIFilterGroup.java:
172)
at
org.opendatakit.aggregate.task.CsvWorkerImpl.generateCsv(CsvWorkerImpl.java:
96)
at
org.opendatakit.aggregate.task.gae.servlet.CsvGeneratorTaskServlet.doGet(CsvGeneratorTaskServlet.java:
96)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:617)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:717)
at org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:
511)
at org.mortbay.jetty.servlet.ServletHandler
$CachedChain.doFilter(ServletHandler.java:1166)
at org.springframework.security.web.FilterChainProxy
$VirtualFilterChain.doFilter(FilterChainProxy.java:368)

Could you please help me to resolve this

Thanks,
Sel

··· On Nov 24, 12:09 am, Mitch S wrote: > I assume this is an AppEngine instance... > (1) Please go to the AppEngine dashboard (viahttps://appengine.google.com/), select the "Permissions" link on the > left sidebar and invite me ( > mitchellsu...@gmail.com ) as a Developer to your AppEngine instance. > (2) Within ODK Aggregate, please add me with Site Admin privileges. > > And let me know what your appengine URL is. > > Thanks, > > Mitch > > On Wed, Nov 23, 2011 at 9:40 AM, Sanjeev Swamy wrote: > > > > > > > > > > > Hi Yaw, > > > I am using ODK Aggregate v 1.0.1 > > > When I tried using ODK Briefcase, I get the following error while > > retrieving data: > > > NOT ALL SUBMISSIONS RETRIEVED: Error fetching list of submissions: > > Fetch of submission download chunk failed. Detailed error: Unexpected > > java.net.SocketTimeoutException: Read timed out > > FAILED. > > > Thanks, > > > Sanjeev > > > On Nov 23, 12:00 pm, Yaw Anokwa wrote: > > > what version of the server are you using? > > > > also, have you tried briefcase? it should be able to do much of what > > > kobo offline processor does. > > > > On Wed, Nov 23, 2011 at 08:46, Sanjeev Swamy wrote: > > > > Hi Guys, > > > > > We have over a thousand xml data that has been uploaded onto the ODK > > > > Aggregate Server. When we try to export to get a .csv file, we get > > > > the following error: > > > > > Failure - abandoned all retry attempts. > > > > > Is this because of an excess of data? Is there a capacity issue with > > > > the server? We have enabled billing for it. > > > > > I tried using the Kobo Offline Processor but the sync doesn't word so > > > > I stored the xml data on my computer and used transcribe. It created > > > > a csv file but all the variables weren't shown. It only showed the > > > > Section names along with combined data. > > > > > Does anyone know of another way to get the csv file? VBA maybe? > > > > > Thanks in advance, > > > > > Sanjeev > > > > > -- > > > > Post: opendatakit@googlegroups.com > > > > Unsubscribe: opendatakit+unsubscribe@googlegroups.com > > > > Options:http://groups.google.com/group/opendatakit?hl=en > > > -- > > Post: opendatakit@googlegroups.com > > Unsubscribe: opendatakit+unsubscribe@googlegroups.com > > Options:http://groups.google.com/group/opendatakit?hl=en > > -- > Mitch Sundt > Software Engineer > University of Washington > mitchellsu...@gmail.com

How much data do you have in your instance (# records if you know; # of GB
from dashboard)?

Mitch

··· On Mon, Nov 28, 2011 at 3:42 AM, res tester wrote:

Hi..

I'm also getting the same error, I'm using google AppEngine instances,
below is the error log

2011-11-28 02:45:59.872

Error for /gae/csvGeneratorTask
java.lang.OutOfMemoryError: Java heap space
at java.nio.ByteBuffer.wrap(ByteBuffer.java:367)
at
java.lang.StringCoding$StringDecoder.decode(StringCoding.java:155)
at java.lang.StringCoding.decode(StringCoding.java:191)
at java.lang.String.(String.java:451)
at

com.google.appengine.repackaged.com.google.io.protocol.ProtocolSupport.toStringUtf8(ProtocolSupport.java:
516)
at

com.google.appengine.repackaged.com.google.io.protocol.ProtocolSupport.toStringUtf8(ProtocolSupport.java:
503)
at com.google.storage.onestore.v3.OnestoreEntity
$Property.getName(OnestoreEntity.java:2989)
at

com.google.appengine.api.datastore.DataTypeTranslator.addPropertyValueToMap(DataTypeTranslator.java:
309)
at

com.google.appengine.api.datastore.DataTypeTranslator.extractIndexedPropertiesFromPb(DataTypeTranslator.java:
217)
at

com.google.appengine.api.datastore.DataTypeTranslator.extractPropertiesFromPb(DataTypeTranslator.java:
238)
at

com.google.appengine.api.datastore.EntityTranslator.createFromPb(EntityTranslator.java:
29)
at

com.google.appengine.api.datastore.QueryResultsSourceImpl.processQueryResult(QueryResultsSourceImpl.java:
153)
at

com.google.appengine.api.datastore.QueryResultsSourceImpl.loadMoreEntities(QueryResultsSourceImpl.java:
86)
at

com.google.appengine.api.datastore.QueryResultsSourceImpl.loadMoreEntities(QueryResultsSourceImpl.java:
71)
at

com.google.appengine.api.datastore.QueryResultIteratorImpl.ensureLoaded(QueryResultIteratorImpl.java:
156)
at

com.google.appengine.api.datastore.QueryResultIteratorImpl.hasNext(QueryResultIteratorImpl.java:
65)
at

org.opendatakit.common.persistence.engine.gae.QueryImpl.fetchResults(QueryImpl.java:
844)
at

org.opendatakit.common.persistence.engine.gae.QueryImpl.chunkFetch(QueryImpl.java:
776)
at

org.opendatakit.common.persistence.engine.gae.QueryImpl.coreExecuteQuery(QueryImpl.java:
1035)
at

org.opendatakit.common.persistence.engine.gae.QueryImpl.executeQuery(QueryImpl.java:
1141)
at

org.opendatakit.aggregate.submission.type.ChoiceSubmissionType.getValueFromEntity(ChoiceSubmissionType.java:
80)
at

org.opendatakit.aggregate.submission.SubmissionSet.buildSubmissionFields(SubmissionSet.java:
481)
at

org.opendatakit.aggregate.submission.SubmissionSet.(SubmissionSet.java:
283)
at
org.opendatakit.aggregate.submission.Submission.(Submission.java:
75)
at

org.opendatakit.aggregate.query.submission.QueryByUIFilterGroup.getResultSubmissions(QueryByUIFilterGroup.java:
172)
at

org.opendatakit.aggregate.task.CsvWorkerImpl.generateCsv(CsvWorkerImpl.java:
96)
at

org.opendatakit.aggregate.task.gae.servlet.CsvGeneratorTaskServlet.doGet(CsvGeneratorTaskServlet.java:
96)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:617)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:717)
at
org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:
511)
at org.mortbay.jetty.servlet.ServletHandler
$CachedChain.doFilter(ServletHandler.java:1166)
at org.springframework.security.web.FilterChainProxy
$VirtualFilterChain.doFilter(FilterChainProxy.java:368)

Could you please help me to resolve this

Thanks,
Sel
On Nov 24, 12:09 am, Mitch S mitchellsu...@gmail.com wrote:

I assume this is an AppEngine instance...
(1) Please go to the AppEngine dashboard (viahttps://
appengine.google.com/), select the "Permissions" link on the
left sidebar and invite me (
mitchellsu...@gmail.com ) as a Developer to your AppEngine instance.
(2) Within ODK Aggregate, please add me with Site Admin privileges.

And let me know what your appengine URL is.

Thanks,

Mitch

On Wed, Nov 23, 2011 at 9:40 AM, Sanjeev Swamy < ssw...@poverty-action.org>wrote:

Hi Yaw,

I am using ODK Aggregate v 1.0.1

When I tried using ODK Briefcase, I get the following error while
retrieving data:

NOT ALL SUBMISSIONS RETRIEVED: Error fetching list of submissions:
Fetch of submission download chunk failed. Detailed error: Unexpected
java.net.SocketTimeoutException: Read timed out
FAILED.

Thanks,

Sanjeev

On Nov 23, 12:00 pm, Yaw Anokwa yano...@gmail.com wrote:

what version of the server are you using?

also, have you tried briefcase? it should be able to do much of what
kobo offline processor does.

On Wed, Nov 23, 2011 at 08:46, Sanjeev Swamy < ssw...@poverty-action.org> wrote:

Hi Guys,

We have over a thousand xml data that has been uploaded onto the
ODK

Aggregate Server. When we try to export to get a .csv file, we get
the following error:

Failure - abandoned all retry attempts.

Is this because of an excess of data? Is there a capacity issue
with

the server? We have enabled billing for it.

I tried using the Kobo Offline Processor but the sync doesn't word
so

I stored the xml data on my computer and used transcribe. It
created

a csv file but all the variables weren't shown. It only showed the
Section names along with combined data.

Does anyone know of another way to get the csv file? VBA maybe?

Thanks in advance,

Sanjeev

--
Post: opendatakit@googlegroups.com
Unsubscribe: opendatakit+unsubscribe@googlegroups.com
Options:http://groups.google.com/group/opendatakit?hl=en

--
Post: opendatakit@googlegroups.com
Unsubscribe: opendatakit+unsubscribe@googlegroups.com
Options:http://groups.google.com/group/opendatakit?hl=en

--
Mitch Sundt
Software Engineer
University of Washington
mitchellsu...@gmail.com

--
Post: opendatakit@googlegroups.com
Unsubscribe: opendatakit+unsubscribe@googlegroups.com
Options: http://groups.google.com/group/opendatakit?hl=en

--
Mitch Sundt
Software Engineer


University of Washington
mitchellsundt@gmail.com

Hi Mitch,

I have more than 2500 records, each record will contain one image data
and around 35 other data types.
Approximate size of each record would be between 600kb - 1.2MB

Overall size would be 1.5 GB

Regards,
Sel

··· On Nov 29, 1:03 am, Mitch Sundt wrote: > How much data do you have in your instance (# records if you know; # of GB > from dashboard)? > > Mitch > > > > On Mon, Nov 28, 2011 at 3:42 AM, res tester wrote: > > Hi.. > > > I'm also getting the same error, I'm using google AppEngine instances, > > below is the error log > > > 2011-11-28 02:45:59.872 > > > Error for /gae/csvGeneratorTask > > java.lang.OutOfMemoryError: Java heap space > > at java.nio.ByteBuffer.wrap(ByteBuffer.java:367) > > at > > java.lang.StringCoding$StringDecoder.decode(StringCoding.java:155) > > at java.lang.StringCoding.decode(StringCoding.java:191) > > at java.lang.String.(String.java:451) > > at > > > com.google.appengine.repackaged.com.google.io.protocol.ProtocolSupport.toStringUtf8(ProtocolSupport.java: > > 516) > > at > > > com.google.appengine.repackaged.com.google.io.protocol.ProtocolSupport.toStringUtf8(ProtocolSupport.java: > > 503) > > at com.google.storage.onestore.v3.OnestoreEntity > > $Property.getName(OnestoreEntity.java:2989) > > at > > > com.google.appengine.api.datastore.DataTypeTranslator.addPropertyValueToMap(DataTypeTranslator.java: > > 309) > > at > > > com.google.appengine.api.datastore.DataTypeTranslator.extractIndexedPropertiesFromPb(DataTypeTranslator.java: > > 217) > > at > > > com.google.appengine.api.datastore.DataTypeTranslator.extractPropertiesFromPb(DataTypeTranslator.java: > > 238) > > at > > > com.google.appengine.api.datastore.EntityTranslator.createFromPb(EntityTranslator.java: > > 29) > > at > > > com.google.appengine.api.datastore.QueryResultsSourceImpl.processQueryResult(QueryResultsSourceImpl.java: > > 153) > > at > > > com.google.appengine.api.datastore.QueryResultsSourceImpl.loadMoreEntities(QueryResultsSourceImpl.java: > > 86) > > at > > > com.google.appengine.api.datastore.QueryResultsSourceImpl.loadMoreEntities(QueryResultsSourceImpl.java: > > 71) > > at > > > com.google.appengine.api.datastore.QueryResultIteratorImpl.ensureLoaded(QueryResultIteratorImpl.java: > > 156) > > at > > > com.google.appengine.api.datastore.QueryResultIteratorImpl.hasNext(QueryResultIteratorImpl.java: > > 65) > > at > > > org.opendatakit.common.persistence.engine.gae.QueryImpl.fetchResults(QueryImpl.java: > > 844) > > at > > > org.opendatakit.common.persistence.engine.gae.QueryImpl.chunkFetch(QueryImpl.java: > > 776) > > at > > > org.opendatakit.common.persistence.engine.gae.QueryImpl.coreExecuteQuery(QueryImpl.java: > > 1035) > > at > > > org.opendatakit.common.persistence.engine.gae.QueryImpl.executeQuery(QueryImpl.java: > > 1141) > > at > > > org.opendatakit.aggregate.submission.type.ChoiceSubmissionType.getValueFromEntity(ChoiceSubmissionType.java: > > 80) > > at > > > org.opendatakit.aggregate.submission.SubmissionSet.buildSubmissionFields(SubmissionSet.java: > > 481) > > at > > > org.opendatakit.aggregate.submission.SubmissionSet.(SubmissionSet.java: > > 283) > > at > > org.opendatakit.aggregate.submission.Submission.(Submission.java: > > 75) > > at > > > org.opendatakit.aggregate.query.submission.QueryByUIFilterGroup.getResultSubmissions(QueryByUIFilterGroup.java: > > 172) > > at > > > org.opendatakit.aggregate.task.CsvWorkerImpl.generateCsv(CsvWorkerImpl.java: > > 96) > > at > > > org.opendatakit.aggregate.task.gae.servlet.CsvGeneratorTaskServlet.doGet(CsvGeneratorTaskServlet.java: > > 96) > > at javax.servlet.http.HttpServlet.service(HttpServlet.java:617) > > at javax.servlet.http.HttpServlet.service(HttpServlet.java:717) > > at > > org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java: > > 511) > > at org.mortbay.jetty.servlet.ServletHandler > > $CachedChain.doFilter(ServletHandler.java:1166) > > at org.springframework.security.web.FilterChainProxy > > $VirtualFilterChain.doFilter(FilterChainProxy.java:368) > > > Could you please help me to resolve this > > > Thanks, > > Sel > > On Nov 24, 12:09 am, Mitch S wrote: > > > I assume this is an AppEngine instance... > > > (1) Please go to the AppEngine dashboard (viahttps:// > > appengine.google.com/), select the "Permissions" link on the > > > left sidebar and invite me ( > > > mitchellsu...@gmail.com ) as a Developer to your AppEngine instance. > > > (2) Within ODK Aggregate, please add me with Site Admin privileges. > > > > And let me know what your appengine URL is. > > > > Thanks, > > > > Mitch > > > > On Wed, Nov 23, 2011 at 9:40 AM, Sanjeev Swamy < ssw...@poverty-action.org>wrote: > > > > > Hi Yaw, > > > > > I am using ODK Aggregate v 1.0.1 > > > > > When I tried using ODK Briefcase, I get the following error while > > > > retrieving data: > > > > > NOT ALL SUBMISSIONS RETRIEVED: Error fetching list of submissions: > > > > Fetch of submission download chunk failed. Detailed error: Unexpected > > > > java.net.SocketTimeoutException: Read timed out > > > > FAILED. > > > > > Thanks, > > > > > Sanjeev > > > > > On Nov 23, 12:00 pm, Yaw Anokwa wrote: > > > > > what version of the server are you using? > > > > > > also, have you tried briefcase? it should be able to do much of what > > > > > kobo offline processor does. > > > > > > On Wed, Nov 23, 2011 at 08:46, Sanjeev Swamy < ssw...@poverty-action.org> wrote: > > > > > > Hi Guys, > > > > > > > We have over a thousand xml data that has been uploaded onto the > > ODK > > > > > > Aggregate Server. When we try to export to get a .csv file, we get > > > > > > the following error: > > > > > > > Failure - abandoned all retry attempts. > > > > > > > Is this because of an excess of data? Is there a capacity issue > > with > > > > > > the server? We have enabled billing for it. > > > > > > > I tried using the Kobo Offline Processor but the sync doesn't word > > so > > > > > > I stored the xml data on my computer and used transcribe. It > > created > > > > > > a csv file but all the variables weren't shown. It only showed the > > > > > > Section names along with combined data. > > > > > > > Does anyone know of another way to get the csv file? VBA maybe? > > > > > > > Thanks in advance, > > > > > > > Sanjeev > > > > > > > -- > > > > > > Post: opendatakit@googlegroups.com > > > > > > Unsubscribe: opendatakit+unsubscribe@googlegroups.com > > > > > > Options:http://groups.google.com/group/opendatakit?hl=en > > > > > -- > > > > Post: opendatakit@googlegroups.com > > > > Unsubscribe: opendatakit+unsubscribe@googlegroups.com > > > > Options:http://groups.google.com/group/opendatakit?hl=en > > > > -- > > > Mitch Sundt > > > Software Engineer > > > University of Washington > > > mitchellsu...@gmail.com > > > -- > > Post: opendatakit@googlegroups.com > > Unsubscribe: opendatakit+unsubscribe@googlegroups.com > > Options:http://groups.google.com/group/opendatakit?hl=en > > -- > Mitch Sundt > Software Engineerhttp://www.OpenDataKit.org > University of Washington > mitchellsu...@gmail.com

some more info on datastore statistics
Breakdown by Property Type
Property Type Size
Blob 773 MBytes
String 15 MBytes
Date/Time 4 MBytes
Integer 2 MBytes
Float 205 KBytes
Boolean 41 KBytes
Text 308 Bytes
Metadata 17 MBytes

Total number of entities Size of all entities
64,570 812 MBytes

··· On Nov 29, 1:03 am, Mitch Sundt wrote: > How much data do you have in your instance (# records if you know; # of GB > from dashboard)? > > Mitch > > > > On Mon, Nov 28, 2011 at 3:42 AM, res tester wrote: > > Hi.. > > > I'm also getting the same error, I'm using google AppEngine instances, > > below is the error log > > > 2011-11-28 02:45:59.872 > > > Error for /gae/csvGeneratorTask > > java.lang.OutOfMemoryError: Java heap space > > at java.nio.ByteBuffer.wrap(ByteBuffer.java:367) > > at > > java.lang.StringCoding$StringDecoder.decode(StringCoding.java:155) > > at java.lang.StringCoding.decode(StringCoding.java:191) > > at java.lang.String.(String.java:451) > > at > > > com.google.appengine.repackaged.com.google.io.protocol.ProtocolSupport.toStringUtf8(ProtocolSupport.java: > > 516) > > at > > > com.google.appengine.repackaged.com.google.io.protocol.ProtocolSupport.toStringUtf8(ProtocolSupport.java: > > 503) > > at com.google.storage.onestore.v3.OnestoreEntity > > $Property.getName(OnestoreEntity.java:2989) > > at > > > com.google.appengine.api.datastore.DataTypeTranslator.addPropertyValueToMap(DataTypeTranslator.java: > > 309) > > at > > > com.google.appengine.api.datastore.DataTypeTranslator.extractIndexedPropertiesFromPb(DataTypeTranslator.java: > > 217) > > at > > > com.google.appengine.api.datastore.DataTypeTranslator.extractPropertiesFromPb(DataTypeTranslator.java: > > 238) > > at > > > com.google.appengine.api.datastore.EntityTranslator.createFromPb(EntityTranslator.java: > > 29) > > at > > > com.google.appengine.api.datastore.QueryResultsSourceImpl.processQueryResult(QueryResultsSourceImpl.java: > > 153) > > at > > > com.google.appengine.api.datastore.QueryResultsSourceImpl.loadMoreEntities(QueryResultsSourceImpl.java: > > 86) > > at > > > com.google.appengine.api.datastore.QueryResultsSourceImpl.loadMoreEntities(QueryResultsSourceImpl.java: > > 71) > > at > > > com.google.appengine.api.datastore.QueryResultIteratorImpl.ensureLoaded(QueryResultIteratorImpl.java: > > 156) > > at > > > com.google.appengine.api.datastore.QueryResultIteratorImpl.hasNext(QueryResultIteratorImpl.java: > > 65) > > at > > > org.opendatakit.common.persistence.engine.gae.QueryImpl.fetchResults(QueryImpl.java: > > 844) > > at > > > org.opendatakit.common.persistence.engine.gae.QueryImpl.chunkFetch(QueryImpl.java: > > 776) > > at > > > org.opendatakit.common.persistence.engine.gae.QueryImpl.coreExecuteQuery(QueryImpl.java: > > 1035) > > at > > > org.opendatakit.common.persistence.engine.gae.QueryImpl.executeQuery(QueryImpl.java: > > 1141) > > at > > > org.opendatakit.aggregate.submission.type.ChoiceSubmissionType.getValueFromEntity(ChoiceSubmissionType.java: > > 80) > > at > > > org.opendatakit.aggregate.submission.SubmissionSet.buildSubmissionFields(SubmissionSet.java: > > 481) > > at > > > org.opendatakit.aggregate.submission.SubmissionSet.(SubmissionSet.java: > > 283) > > at > > org.opendatakit.aggregate.submission.Submission.(Submission.java: > > 75) > > at > > > org.opendatakit.aggregate.query.submission.QueryByUIFilterGroup.getResultSubmissions(QueryByUIFilterGroup.java: > > 172) > > at > > > org.opendatakit.aggregate.task.CsvWorkerImpl.generateCsv(CsvWorkerImpl.java: > > 96) > > at > > > org.opendatakit.aggregate.task.gae.servlet.CsvGeneratorTaskServlet.doGet(CsvGeneratorTaskServlet.java: > > 96) > > at javax.servlet.http.HttpServlet.service(HttpServlet.java:617) > > at javax.servlet.http.HttpServlet.service(HttpServlet.java:717) > > at > > org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java: > > 511) > > at org.mortbay.jetty.servlet.ServletHandler > > $CachedChain.doFilter(ServletHandler.java:1166) > > at org.springframework.security.web.FilterChainProxy > > $VirtualFilterChain.doFilter(FilterChainProxy.java:368) > > > Could you please help me to resolve this > > > Thanks, > > Sel > > On Nov 24, 12:09 am, Mitch S wrote: > > > I assume this is an AppEngine instance... > > > (1) Please go to the AppEngine dashboard (viahttps:// > > appengine.google.com/), select the "Permissions" link on the > > > left sidebar and invite me ( > > > mitchellsu...@gmail.com ) as a Developer to your AppEngine instance. > > > (2) Within ODK Aggregate, please add me with Site Admin privileges. > > > > And let me know what your appengine URL is. > > > > Thanks, > > > > Mitch > > > > On Wed, Nov 23, 2011 at 9:40 AM, Sanjeev Swamy < ssw...@poverty-action.org>wrote: > > > > > Hi Yaw, > > > > > I am using ODK Aggregate v 1.0.1 > > > > > When I tried using ODK Briefcase, I get the following error while > > > > retrieving data: > > > > > NOT ALL SUBMISSIONS RETRIEVED: Error fetching list of submissions: > > > > Fetch of submission download chunk failed. Detailed error: Unexpected > > > > java.net.SocketTimeoutException: Read timed out > > > > FAILED. > > > > > Thanks, > > > > > Sanjeev > > > > > On Nov 23, 12:00 pm, Yaw Anokwa wrote: > > > > > what version of the server are you using? > > > > > > also, have you tried briefcase? it should be able to do much of what > > > > > kobo offline processor does. > > > > > > On Wed, Nov 23, 2011 at 08:46, Sanjeev Swamy < ssw...@poverty-action.org> wrote: > > > > > > Hi Guys, > > > > > > > We have over a thousand xml data that has been uploaded onto the > > ODK > > > > > > Aggregate Server. When we try to export to get a .csv file, we get > > > > > > the following error: > > > > > > > Failure - abandoned all retry attempts. > > > > > > > Is this because of an excess of data? Is there a capacity issue > > with > > > > > > the server? We have enabled billing for it. > > > > > > > I tried using the Kobo Offline Processor but the sync doesn't word > > so > > > > > > I stored the xml data on my computer and used transcribe. It > > created > > > > > > a csv file but all the variables weren't shown. It only showed the > > > > > > Section names along with combined data. > > > > > > > Does anyone know of another way to get the csv file? VBA maybe? > > > > > > > Thanks in advance, > > > > > > > Sanjeev > > > > > > > -- > > > > > > Post: opendatakit@googlegroups.com > > > > > > Unsubscribe: opendatakit+unsubscribe@googlegroups.com > > > > > > Options:http://groups.google.com/group/opendatakit?hl=en > > > > > -- > > > > Post: opendatakit@googlegroups.com > > > > Unsubscribe: opendatakit+unsubscribe@googlegroups.com > > > > Options:http://groups.google.com/group/opendatakit?hl=en > > > > -- > > > Mitch Sundt > > > Software Engineer > > > University of Washington > > > mitchellsu...@gmail.com > > > -- > > Post: opendatakit@googlegroups.com > > Unsubscribe: opendatakit+unsubscribe@googlegroups.com > > Options:http://groups.google.com/group/opendatakit?hl=en > > -- > Mitch Sundt > Software Engineerhttp://www.OpenDataKit.org > University of Washington > mitchellsu...@gmail.com

Thanks. I'm investigating this now.

Mitch

··· On Mon, Nov 28, 2011 at 1:13 PM, res tester wrote:

Hi Mitch,

I have more than 2500 records, each record will contain one image data
and around 35 other data types.
Approximate size of each record would be between 600kb - 1.2MB

Overall size would be 1.5 GB

Regards,
Sel

On Nov 29, 1:03 am, Mitch Sundt msu...@cs.washington.edu wrote:

How much data do you have in your instance (# records if you know; # of
GB
from dashboard)?

Mitch

On Mon, Nov 28, 2011 at 3:42 AM, res tester res.test...@gmail.com wrote:

Hi..

I'm also getting the same error, I'm using google AppEngine instances,
below is the error log

2011-11-28 02:45:59.872

Error for /gae/csvGeneratorTask
java.lang.OutOfMemoryError: Java heap space
at java.nio.ByteBuffer.wrap(ByteBuffer.java:367)
at
java.lang.StringCoding$StringDecoder.decode(StringCoding.java:155)
at java.lang.StringCoding.decode(StringCoding.java:191)
at java.lang.String.(String.java:451)
at

com.google.appengine.repackaged.com.google.io.protocol.ProtocolSupport.toStringUtf8(ProtocolSupport.java:

  1. at

com.google.appengine.repackaged.com.google.io.protocol.ProtocolSupport.toStringUtf8(ProtocolSupport.java:

  1. at com.google.storage.onestore.v3.OnestoreEntity
    $Property.getName(OnestoreEntity.java:2989)
    at

com.google.appengine.api.datastore.DataTypeTranslator.addPropertyValueToMap(DataTypeTranslator.java:

  1. at

com.google.appengine.api.datastore.DataTypeTranslator.extractIndexedPropertiesFromPb(DataTypeTranslator.java:

  1. at

com.google.appengine.api.datastore.DataTypeTranslator.extractPropertiesFromPb(DataTypeTranslator.java:

  1. at

com.google.appengine.api.datastore.EntityTranslator.createFromPb(EntityTranslator.java:

  1. at

com.google.appengine.api.datastore.QueryResultsSourceImpl.processQueryResult(QueryResultsSourceImpl.java:

  1. at

com.google.appengine.api.datastore.QueryResultsSourceImpl.loadMoreEntities(QueryResultsSourceImpl.java:

  1. at

com.google.appengine.api.datastore.QueryResultsSourceImpl.loadMoreEntities(QueryResultsSourceImpl.java:

  1. at

com.google.appengine.api.datastore.QueryResultIteratorImpl.ensureLoaded(QueryResultIteratorImpl.java:

  1. at

com.google.appengine.api.datastore.QueryResultIteratorImpl.hasNext(QueryResultIteratorImpl.java:

  1. at

org.opendatakit.common.persistence.engine.gae.QueryImpl.fetchResults(QueryImpl.java:

  1. at

org.opendatakit.common.persistence.engine.gae.QueryImpl.chunkFetch(QueryImpl.java:

  1. at

org.opendatakit.common.persistence.engine.gae.QueryImpl.coreExecuteQuery(QueryImpl.java:

  1. at

org.opendatakit.common.persistence.engine.gae.QueryImpl.executeQuery(QueryImpl.java:

  1. at

org.opendatakit.aggregate.submission.type.ChoiceSubmissionType.getValueFromEntity(ChoiceSubmissionType.java:

  1. at

org.opendatakit.aggregate.submission.SubmissionSet.buildSubmissionFields(SubmissionSet.java:

  1. at

org.opendatakit.aggregate.submission.SubmissionSet.(SubmissionSet.java:

  1. at
    org.opendatakit.aggregate.submission.Submission.(Submission.java:
  2. at

org.opendatakit.aggregate.query.submission.QueryByUIFilterGroup.getResultSubmissions(QueryByUIFilterGroup.java:

  1. at

org.opendatakit.aggregate.task.CsvWorkerImpl.generateCsv(CsvWorkerImpl.java:

  1. at

org.opendatakit.aggregate.task.gae.servlet.CsvGeneratorTaskServlet.doGet(CsvGeneratorTaskServlet.java:

  1. at javax.servlet.http.HttpServlet.service(HttpServlet.java:617)
    at javax.servlet.http.HttpServlet.service(HttpServlet.java:717)
    at
    org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:
  2. at org.mortbay.jetty.servlet.ServletHandler
    $CachedChain.doFilter(ServletHandler.java:1166)
    at org.springframework.security.web.FilterChainProxy
    $VirtualFilterChain.doFilter(FilterChainProxy.java:368)

Could you please help me to resolve this

Thanks,
Sel
On Nov 24, 12:09 am, Mitch S mitchellsu...@gmail.com wrote:

I assume this is an AppEngine instance...
(1) Please go to the AppEngine dashboard (viahttps://
appengine.google.com/), select the "Permissions" link on the
left sidebar and invite me (
mitchellsu...@gmail.com ) as a Developer to your AppEngine instance.
(2) Within ODK Aggregate, please add me with Site Admin privileges.

And let me know what your appengine URL is.

Thanks,

Mitch

On Wed, Nov 23, 2011 at 9:40 AM, Sanjeev Swamy < ssw...@poverty-action.org>wrote:

Hi Yaw,

I am using ODK Aggregate v 1.0.1

When I tried using ODK Briefcase, I get the following error while
retrieving data:

NOT ALL SUBMISSIONS RETRIEVED: Error fetching list of submissions:
Fetch of submission download chunk failed. Detailed error:
Unexpected

java.net.SocketTimeoutException: Read timed out
FAILED.

Thanks,

Sanjeev

On Nov 23, 12:00 pm, Yaw Anokwa yano...@gmail.com wrote:

what version of the server are you using?

also, have you tried briefcase? it should be able to do much of
what

kobo offline processor does.

On Wed, Nov 23, 2011 at 08:46, Sanjeev Swamy < ssw...@poverty-action.org> wrote:

Hi Guys,

We have over a thousand xml data that has been uploaded onto
the

ODK

Aggregate Server. When we try to export to get a .csv file,
we get

the following error:

Failure - abandoned all retry attempts.

Is this because of an excess of data? Is there a capacity issue
with

the server? We have enabled billing for it.

I tried using the Kobo Offline Processor but the sync doesn't
word

so

I stored the xml data on my computer and used transcribe. It
created

a csv file but all the variables weren't shown. It only
showed the

Section names along with combined data.

Does anyone know of another way to get the csv file? VBA maybe?

Thanks in advance,

Sanjeev

--
Post: opendatakit@googlegroups.com
Unsubscribe: opendatakit+unsubscribe@googlegroups.com
Options:http://groups.google.com/group/opendatakit?hl=en

--
Post: opendatakit@googlegroups.com
Unsubscribe: opendatakit+unsubscribe@googlegroups.com
Options:http://groups.google.com/group/opendatakit?hl=en

--
Mitch Sundt
Software Engineer
University of Washington
mitchellsu...@gmail.com

--
Post: opendatakit@googlegroups.com
Unsubscribe: opendatakit+unsubscribe@googlegroups.com
Options:http://groups.google.com/group/opendatakit?hl=en

--
Mitch Sundt
Software Engineerhttp://www.OpenDataKit.org
University of Washington
mitchellsu...@gmail.com

--
Post: opendatakit@googlegroups.com
Unsubscribe: opendatakit+unsubscribe@googlegroups.com
Options: http://groups.google.com/group/opendatakit?hl=en

--
Mitch Sundt
Software Engineer
University of Washington
mitchellsundt@gmail.com