My team is conducting five-hour structured observation in which they record when a child touches food, soil, water etc. After they have selected the object, a questions about that object open. Each event of touching something is recorded as a separate event in a loop. The result is a loop with 100-200 separate events.
One major problem is that during data collection ODK Collect 1.2 stalls. Enumerators cannot move forward or backwards to past or previous questions. Then Collect closes without an error message. Afterwards, they cannot find the data anywhere on the tablet, despite ODK Collect 1.2 reportedly saving after each swipe. This problem has caused the deletion of 10+ hours of data so any advice would be happily received.
A minor issue is that if the file has more than 120 events in the loop, it is usually not able to upload to Aggregate (my frontend instance class is F1 - one time upload was successful running F2 but one time it was not and both loops contained approximately 190 events). I get a Internal Server Error:500 (picture attached).
We recently contributed a variety of improvements to the auto-recovery
mechanisms in ODK Collect, from our SurveyCTO version of Collect. We have
also contributed code to dramatically reduce memory requirements (which you
may be running into when you "stall"). I imagine that those contributions
will be in the next Collect release, but there is no date for that as yet.
As for your Internal Server Errors, you'd need to look into the server logs
and diagnose what's going on. The problem could relate to your Tomcat or
MySQL settings, presuming that's what you're running. Your server
manager/programmer may need to sort that out, though if you post specific
errors from your logs then folks in this group may be able to help.
Best,
Chris
···
On Wed, Mar 12, 2014 at 8:35 AM, wrote:
My team is conducting five-hour structured observation in which they
record when a child touches food, soil, water etc. After they have selected
the object, a questions about that object open. Each event of touching
something is recorded as a separate event in a loop. The result is a loop
with 100-200 separate events.
One major problem is that during data collection ODK Collect 1.2 stalls.
Enumerators cannot move forward or backwards to past or previous questions.
Then Collect closes without an error message. Afterwards, they cannot find
the data anywhere on the tablet, despite ODK Collect 1.2 reportedly saving
after each swipe. This problem has caused the deletion of 10+ hours of data
so any advice would be happily received.
A minor issue is that if the file has more than 120 events in the loop, it
is usually not able to upload to Aggregate (my frontend instance class is
F1 - one time upload was successful running F2 but one time it was not and
both loops contained approximately 190 events). I get a Internal Server
Error:500 (picture attached).
For the stalling issue, is the problem the total magnitude of data held by collect or the amount of data in one survey? Would a solution be to break the survey into smaller pieces? This is easily done. If you have an idea you think would work better, please let me know. I'm guessing this will also solve the upload issue, which I think it due to timing out.
I attached the error log from three failed attempts to submit. Unfortunately, I am the server manager/programmer! We are using Aggregate on Google's app engine so I do not think we are using Tomcat or MySQL.
Hopefully somebody will have an idea about the GAE issue.
For memory, the issue is annoyingly device- and survey-specific. Devices
decide how much memory to allocate to each app, and then the memory usage
depends on the survey in-memory. If you use lots of repeats or have lots of
multiple-choice options, those can chew up memory. Come to think of it, if
you're using Collect 1.2 then just upgrading to 1.4 may help; at some point
in recent memory, the team added a flag to ask the device for more memory
("largeHeap"), and that may help.
Best,
Chris
···
On Wed, Mar 12, 2014 at 2:18 PM, wrote:
Hi Chris,
Thanks for your incredibly quick reply.
For the stalling issue, is the problem the total magnitude of data held by
collect or the amount of data in one survey? Would a solution be to break
the survey into smaller pieces? This is easily done. If you have an idea
you think would work better, please let me know. I'm guessing this will
also solve the upload issue, which I think it due to timing out.
I attached the error log from three failed attempts to submit.
Unfortunately, I am the server manager/programmer! We are using Aggregate
on Google's app engine so I do not think we are using Tomcat or MySQL.
Do you think that splitting the form into smaller pieces would help or not?
Then only a small chunk of repeats would be in an "active" form and maybe
the "saved and finalize forms" wouldn't take up the same memory space?
Either way, I'll upgrade to Collect 1.4.
Thanks,
Laura
···
On Thu, Mar 13, 2014 at 12:06 AM, Christopher Robert wrote:
Hi Laura,
Hopefully somebody will have an idea about the GAE issue.
For memory, the issue is annoyingly device- and survey-specific. Devices
decide how much memory to allocate to each app, and then the memory usage
depends on the survey in-memory. If you use lots of repeats or have lots of
multiple-choice options, those can chew up memory. Come to think of it, if
you're using Collect 1.2 then just upgrading to 1.4 may help; at some point
in recent memory, the team added a flag to ask the device for more memory
("largeHeap"), and that may help.
For the stalling issue, is the problem the total magnitude of data held
by collect or the amount of data in one survey? Would a solution be to
break the survey into smaller pieces? This is easily done. If you have an
idea you think would work better, please let me know. I'm guessing this
will also solve the upload issue, which I think it due to timing out.
I attached the error log from three failed attempts to submit.
Unfortunately, I am the server manager/programmer! We are using Aggregate
on Google's app engine so I do not think we are using Tomcat or MySQL.
Thanks again.
--
You received this message because you are subscribed to the Google Groups
"ODK Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to opendatakit-developers+unsubscribe@googlegroups.com.
Yes, splitting the form up will help if the issue is memory-related. It's
hard to know whether it is or is not memory related, but it's something to
try. (Finally in our next SurveyCTO version of Collect it'll be easy to
submit crash reports. Without seeing details about a crash, it's hard to
diagnose exactly what's going on.)
Do you think that splitting the form into smaller pieces would help or
not? Then only a small chunk of repeats would be in an "active" form and
maybe the "saved and finalize forms" wouldn't take up the same memory space?
Either way, I'll upgrade to Collect 1.4.
Hopefully somebody will have an idea about the GAE issue.
For memory, the issue is annoyingly device- and survey-specific. Devices
decide how much memory to allocate to each app, and then the memory usage
depends on the survey in-memory. If you use lots of repeats or have lots of
multiple-choice options, those can chew up memory. Come to think of it, if
you're using Collect 1.2 then just upgrading to 1.4 may help; at some point
in recent memory, the team added a flag to ask the device for more memory
("largeHeap"), and that may help.
For the stalling issue, is the problem the total magnitude of data held
by collect or the amount of data in one survey? Would a solution be to break
the survey into smaller pieces? This is easily done. If you have an idea you
think would work better, please let me know. I'm guessing this will also
solve the upload issue, which I think it due to timing out.
I attached the error log from three failed attempts to submit.
Unfortunately, I am the server manager/programmer! We are using Aggregate on
Google's app engine so I do not think we are using Tomcat or MySQL.
Thanks again.
--
You received this message because you are subscribed to the Google
Groups "ODK Developers" group.
To unsubscribe from this group and stop receiving emails from it, send
an email to opendatakit-developers+unsubscribe@googlegroups.com.
Yes, splitting the form up will help if the issue is memory-related. It's
hard to know whether it is or is not memory related, but it's something to
try. (Finally in our next SurveyCTO version of Collect it'll be easy to
submit crash reports. Without seeing details about a crash, it's hard to
diagnose exactly what's going on.)
Best,
Chris
···
On Wed, Mar 12, 2014 at 3:02 PM, Laura H Kwong wrote:
Hi Chris,
Do you think that splitting the form into smaller pieces would help or
not? Then only a small chunk of repeats would be in an "active" form and
maybe the "saved and finalize forms" wouldn't take up the same memory
space? Either way, I'll upgrade to Collect 1.4.
Thanks,
Laura
On Thu, Mar 13, 2014 at 12:06 AM, Christopher Robert < crobert@surveycto.com> wrote:
Hi Laura,
Hopefully somebody will have an idea about the GAE issue.
For memory, the issue is annoyingly device- and survey-specific. Devices
decide how much memory to allocate to each app, and then the memory usage
depends on the survey in-memory. If you use lots of repeats or have lots of
multiple-choice options, those can chew up memory. Come to think of it, if
you're using Collect 1.2 then just upgrading to 1.4 may help; at some point
in recent memory, the team added a flag to ask the device for more memory
("largeHeap"), and that may help.
For the stalling issue, is the problem the total magnitude of data held
by collect or the amount of data in one survey? Would a solution be to
break the survey into smaller pieces? This is easily done. If you have an
idea you think would work better, please let me know. I'm guessing this
will also solve the upload issue, which I think it due to timing out.
I attached the error log from three failed attempts to submit.
Unfortunately, I am the server manager/programmer! We are using Aggregate
on Google's app engine so I do not think we are using Tomcat or MySQL.
Thanks again.
--
You received this message because you are subscribed to the Google
Groups "ODK Developers" group.
To unsubscribe from this group and stop receiving emails from it, send
an email to opendatakit-developers+unsubscribe@googlegroups.com.
Yes, we have $5.00 / day quota and regularly only use 20 cents or less, 15
cents on datastore read operations, only 3 cents on datastore write
operations, and 1 cent on datastore stored data yesterday from when the
error logs were made.
···
On Thu, Mar 13, 2014 at 12:50 AM, Yaw Anokwa wrote:
Laura,
And for anything related to GAE, make sure you have enough quota. Can
you confirm that you do?
Yaw
Need ODK services? http://nafundi.com provides form design, server
setup, professional support, and software development for ODK.
Yes, splitting the form up will help if the issue is memory-related. It's
hard to know whether it is or is not memory related, but it's something
to
try. (Finally in our next SurveyCTO version of Collect it'll be easy to
submit crash reports. Without seeing details about a crash, it's hard to
diagnose exactly what's going on.)
Do you think that splitting the form into smaller pieces would help or
not? Then only a small chunk of repeats would be in an "active" form and
maybe the "saved and finalize forms" wouldn't take up the same memory
space?
Either way, I'll upgrade to Collect 1.4.
Hopefully somebody will have an idea about the GAE issue.
For memory, the issue is annoyingly device- and survey-specific.
Devices
decide how much memory to allocate to each app, and then the memory
usage
depends on the survey in-memory. If you use lots of repeats or have
lots of
multiple-choice options, those can chew up memory. Come to think of
it, if
you're using Collect 1.2 then just upgrading to 1.4 may help; at some
point
in recent memory, the team added a flag to ask the device for more
memory
("largeHeap"), and that may help.
For the stalling issue, is the problem the total magnitude of data
held
by collect or the amount of data in one survey? Would a solution be
to break
the survey into smaller pieces? This is easily done. If you have an
idea you
think would work better, please let me know. I'm guessing this will
also
solve the upload issue, which I think it due to timing out.
I attached the error log from three failed attempts to submit.
Unfortunately, I am the server manager/programmer! We are using
Aggregate on
Google's app engine so I do not think we are using Tomcat or MySQL.
Thanks again.
--
You received this message because you are subscribed to the Google
Groups "ODK Developers" group.
To unsubscribe from this group and stop receiving emails from it, send
an email to opendatakit-developers+unsubscribe@googlegroups.com.
Yes, we have $5.00 / day quota and regularly only use 20 cents or less, 15
cents on datastore read operations, only 3 cents on datastore write
operations, and 1 cent on datastore stored data yesterday from when the
error logs were made.
Yes, splitting the form up will help if the issue is memory-related.
It's
hard to know whether it is or is not memory related, but it's something
to
try. (Finally in our next SurveyCTO version of Collect it'll be easy to
submit crash reports. Without seeing details about a crash, it's hard to
diagnose exactly what's going on.)
Do you think that splitting the form into smaller pieces would help or
not? Then only a small chunk of repeats would be in an "active" form
and
maybe the "saved and finalize forms" wouldn't take up the same memory
space?
Either way, I'll upgrade to Collect 1.4.
Hopefully somebody will have an idea about the GAE issue.
For memory, the issue is annoyingly device- and survey-specific.
Devices
decide how much memory to allocate to each app, and then the memory
usage
depends on the survey in-memory. If you use lots of repeats or have
lots of
multiple-choice options, those can chew up memory. Come to think of
it, if
you're using Collect 1.2 then just upgrading to 1.4 may help; at some
point
in recent memory, the team added a flag to ask the device for more
memory
("largeHeap"), and that may help.
For the stalling issue, is the problem the total magnitude of data
held
by collect or the amount of data in one survey? Would a solution be
to break
the survey into smaller pieces? This is easily done. If you have an
idea you
think would work better, please let me know. I'm guessing this will
also
solve the upload issue, which I think it due to timing out.
I attached the error log from three failed attempts to submit.
Unfortunately, I am the server manager/programmer! We are using
Aggregate on
Google's app engine so I do not think we are using Tomcat or MySQL.
Thanks again.
--
You received this message because you are subscribed to the Google
Groups "ODK Developers" group.
To unsubscribe from this group and stop receiving emails from it,
send
an email to opendatakit-developers+unsubscribe@googlegroups.com.
You are hitting a 60-second processing limit during the submission. This
is a hard limit dictated by Google -
The most likely cause of this is a slow internet connection. If you submit
via Satellite phone, or 3G, it could take so long to send the submission to
the server, that the server won't have fully received it and processed it
before the 60-second limit is exceeded. You may have better success just
leaving the surveys on the phone, then submitting them when you get to a
good, fast, wifi connection.
The second possibility, since you have such a large form, is that the
server is running low on available memory. This can cause it to churn and
waste time searching for every scrap of free space.
If you suspect the second possibility, you can go to your appengine
dashboard, select "Application Settings" and change the "Frontend Instance
Class" to the next higher size. This will double the memory (you should
never need to go higher than that).
This will run down your free quota 2x faster, so if you don't see an
improvement, set it back down to the lowest size.
Attached is a screen shot showing what to look for.
If neither of these address the issue, you would need to set up and manage
your own server, where you can adjust the processing limits as needed.
Yes, we have $5.00 / day quota and regularly only use 20 cents or less,
15
cents on datastore read operations, only 3 cents on datastore write
operations, and 1 cent on datastore stored data yesterday from when the
error logs were made.
Yes, splitting the form up will help if the issue is memory-related.
It's
hard to know whether it is or is not memory related, but it's
something
to
try. (Finally in our next SurveyCTO version of Collect it'll be easy
to
submit crash reports. Without seeing details about a crash, it's hard
to
diagnose exactly what's going on.)
Do you think that splitting the form into smaller pieces would help
or
not? Then only a small chunk of repeats would be in an "active" form
and
maybe the "saved and finalize forms" wouldn't take up the same memory
space?
Either way, I'll upgrade to Collect 1.4.
Hopefully somebody will have an idea about the GAE issue.
For memory, the issue is annoyingly device- and survey-specific.
Devices
decide how much memory to allocate to each app, and then the memory
usage
depends on the survey in-memory. If you use lots of repeats or have
lots of
multiple-choice options, those can chew up memory. Come to think of
it, if
you're using Collect 1.2 then just upgrading to 1.4 may help; at
some
point
in recent memory, the team added a flag to ask the device for more
memory
("largeHeap"), and that may help.
For the stalling issue, is the problem the total magnitude of data
held
by collect or the amount of data in one survey? Would a solution be
to break
the survey into smaller pieces? This is easily done. If you have an
idea you
think would work better, please let me know. I'm guessing this will
also
solve the upload issue, which I think it due to timing out.
I attached the error log from three failed attempts to submit.
Unfortunately, I am the server manager/programmer! We are using
Aggregate on
Google's app engine so I do not think we are using Tomcat or MySQL.
Thanks again.
--
You received this message because you are subscribed to the Google
Groups "ODK Developers" group.
To unsubscribe from this group and stop receiving emails from it,
send
an email to opendatakit-developers+unsubscribe@googlegroups.com.
Thanks, Mitch. I've already tried increasing the quota to F2 but it didn't
seem to help. We working in Bangladesh and the internet isn't great so I'm
guessing that the 60-second limit is the problem. Good to know that this is
a hard limit.
Laura
···
On Fri, Mar 14, 2014 at 12:36 AM, Mitch Sundt wrote:
The most likely cause of this is a slow internet connection. If you submit
via Satellite phone, or 3G, it could take so long to send the submission to
the server, that the server won't have fully received it and processed it
before the 60-second limit is exceeded. You may have better success just
leaving the surveys on the phone, then submitting them when you get to a
good, fast, wifi connection.
The second possibility, since you have such a large form, is that the
server is running low on available memory. This can cause it to churn and
waste time searching for every scrap of free space.
If you suspect the second possibility, you can go to your appengine
dashboard, select "Application Settings" and change the "Frontend Instance
Class" to the next higher size. This will double the memory (you should
never need to go higher than that).
This will run down your free quota 2x faster, so if you don't see an
improvement, set it back down to the lowest size.
Attached is a screen shot showing what to look for.
If neither of these address the issue, you would need to set up and manage
your own server, where you can adjust the processing limits as needed.
Yes, we have $5.00 / day quota and regularly only use 20 cents or less,
15
cents on datastore read operations, only 3 cents on datastore write
operations, and 1 cent on datastore stored data yesterday from when the
error logs were made.
Yes, splitting the form up will help if the issue is memory-related.
It's
hard to know whether it is or is not memory related, but it's
something
to
try. (Finally in our next SurveyCTO version of Collect it'll be easy
to
submit crash reports. Without seeing details about a crash, it's
hard to
diagnose exactly what's going on.)
Do you think that splitting the form into smaller pieces would help
or
not? Then only a small chunk of repeats would be in an "active" form
and
maybe the "saved and finalize forms" wouldn't take up the same
memory
space?
Either way, I'll upgrade to Collect 1.4.
Hopefully somebody will have an idea about the GAE issue.
For memory, the issue is annoyingly device- and survey-specific.
Devices
decide how much memory to allocate to each app, and then the memory
usage
depends on the survey in-memory. If you use lots of repeats or have
lots of
multiple-choice options, those can chew up memory. Come to think of
it, if
you're using Collect 1.2 then just upgrading to 1.4 may help; at
some
point
in recent memory, the team added a flag to ask the device for more
memory
("largeHeap"), and that may help.
For the stalling issue, is the problem the total magnitude of data
held
by collect or the amount of data in one survey? Would a solution
be
to break
the survey into smaller pieces? This is easily done. If you have
an
idea you
think would work better, please let me know. I'm guessing this
will
also
solve the upload issue, which I think it due to timing out.
I attached the error log from three failed attempts to submit.
Unfortunately, I am the server manager/programmer! We are using
Aggregate on
Google's app engine so I do not think we are using Tomcat or
MySQL.
Thanks again.
--
You received this message because you are subscribed to the Google
Groups "ODK Developers" group.
To unsubscribe from this group and stop receiving emails from it,
send
an email to opendatakit-developers+unsubscribe@googlegroups.com.
