Sending stuck in a loop

Development
1

What would make the sending of a form with media files get stuck in a loop trying over and over to get past sending that n'th media file in a form?

What I have tried...

  1. closing and restarting SurveyCTO and restarting the send
  2. checked my connection --- I can browse the internet from the device
  3. waiting enough time --- my phone data usage shows that the app has sent 4x the data required to send the form
  4. make sure I am sending only one form

After all this only 650 of the media files have successfully send and 350 remain unsent.

The phone's data traffic monitor shows consistent increase in data movement. So I can deduce that SurveyCTO is attempting to send the same n'th file over and over again... but having to resend it because of some retry logic.

One thought is that one media file is 20 percent larger than the others and the time stamp has it about 650 down the list of media files to upload. If the send goes by time stamp order that would put it right about the time when the process is getting stuck in a loop. Could it be that that 12MB file size is getting stuck in a loop of trying to upload?

If so... can I test this by removing or replacing that file with one of the same name but smaller file size and restart the send to see if that corrects the looping? As long as replace the suspect media file with one smaller but the same name will that disturb the marked completed form?

Many thanks!
Bob

2

Bob,

Sounds like you should email the guys at SurveyCTO...

Yaw

··· -- Need ODK services? http://nafundi.com provides form design, server setup, professional support, and software development for ODK.

On Tue, Jul 1, 2014 at 6:10 PM, bobachgill@gmail.com wrote:

What would make the sending of a form with media files get stuck in a loop trying over and over to get past sending that n'th media file in a form?

What I have tried...

  1. closing and restarting SurveyCTO and restarting the send
  2. checked my connection --- I can browse the internet from the device
  3. waiting enough time --- my phone data usage shows that the app has sent 4x the data required to send the form
  4. make sure I am sending only one form

After all this only 650 of the media files have successfully send and 350 remain unsent.

The phone's data traffic monitor shows consistent increase in data movement. So I can deduce that SurveyCTO is attempting to send the same n'th file over and over again... but having to resend it because of some retry logic.

One thought is that one media file is 20 percent larger than the others and the time stamp has it about 650 down the list of media files to upload. If the send goes by time stamp order that would put it right about the time when the process is getting stuck in a loop. Could it be that that 12MB file size is getting stuck in a loop of trying to upload?

If so... can I test this by removing or replacing that file with one of the same name but smaller file size and restart the send to see if that corrects the looping? As long as replace the suspect media file with one smaller but the same name will that disturb the marked completed form?

Many thanks!
Bob

--
You received this message because you are subscribed to the Google Groups "ODK Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to opendatakit-developers+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

3

One thought is that one media file is 20 percent larger than the others and the time stamp has it about 650 down the list of media files to upload. If the send goes by time stamp order that would put it right about the time when the process is getting stuck in a loop. Could it be that that 12MB file size is getting stuck in a loop of trying to upload?

If so... can I test this by removing or replacing that file with one of the same name but smaller file size and restart the send to see if that corrects the looping? As long as replace the suspect media file with one smaller but the same name will that disturb the marked completed form?

Well, I tried making that larger video smaller and restarting the send... but still stuck in a loop. :frowning:

The other remaining videos are between 3-8MB

So back to the question... why would the upload of the n'th video file keep retrying?

Bob

4

Just for grins...
From what I see here in China... when the export flood gate is opened... smart phones in the world will all be stamped "made in China". They are using their own population to test them... the $100 to $200 models looking really good!

5

I realize that my questions may seem out place on this forum since I am using a derivative of ODK... but I escalate my question from the SurveyCTO help desk back to this forum when it appears that the problem might be germane to the base line ODK code.

6

Hi Yaw,

Yes, sorry for the trouble. We are working with Bob via our support system.
As he says, he is sending a form from China with over 1,500 multi-megabyte
media attachments. This may be a case where sending via the server is just
not going to work as there may be numerous challenges. We're working to
support him, but his is obviously a challenging and very unusual case. (He
also gives us only a few hours to resolve something before coming to this
forum.)

Best,

Chris

··· On Tue, Jul 1, 2014 at 9:12 PM, Yaw Anokwa wrote:

Bob,

Sounds like you should email the guys at SurveyCTO...

Yaw

Need ODK services? http://nafundi.com provides form design, server
setup, professional support, and software development for ODK.

On Tue, Jul 1, 2014 at 6:10 PM, bobachgill@gmail.com wrote:

What would make the sending of a form with media files get stuck in a
loop trying over and over to get past sending that n'th media file in a
form?

What I have tried...

  1. closing and restarting SurveyCTO and restarting the send
  2. checked my connection --- I can browse the internet from the device
  3. waiting enough time --- my phone data usage shows that the app has
    sent 4x the data required to send the form
  4. make sure I am sending only one form

After all this only 650 of the media files have successfully send and
350 remain unsent.

The phone's data traffic monitor shows consistent increase in data
movement. So I can deduce that SurveyCTO is attempting to send the same
n'th file over and over again... but having to resend it because of some
retry logic.

One thought is that one media file is 20 percent larger than the others
and the time stamp has it about 650 down the list of media files to upload.
If the send goes by time stamp order that would put it right about the
time when the process is getting stuck in a loop. Could it be that that
12MB file size is getting stuck in a loop of trying to upload?

If so... can I test this by removing or replacing that file with one of
the same name but smaller file size and restart the send to see if that
corrects the looping? As long as replace the suspect media file with one
smaller but the same name will that disturb the marked completed form?

Many thanks!
Bob

--
You received this message because you are subscribed to the Google
Groups "ODK Developers" group.
To unsubscribe from this group and stop receiving emails from it, send
an email to opendatakit-developers+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups
"ODK Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to opendatakit-developers+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

7

Long term, this would be a useful discussion to open up with all the
OpenRosa server implementers -- adding a mechanism to the protocol to
obtain the list of media attachments already uploaded to the server, along
with their md5 hashes.

Short term, if you are communicating with an ODK Aggregate 1.x-derived
server, the code change to ODK Collect would be:

Obtain list of media attachment files for the submission from the device SD
card / external storage location.

Issue Head request to establish authentication / authorization for session.

If attempting to re-send a submission that previously failed, issue an ODK
Briefcase http://code.google.com/p/opendatakit/wiki/BriefcaseAggregateAPI
GET view/downloadSubmission request to the server.

If this is successful, filter the list of media attachments to exclude
those that the server reports as having been successfully uploaded.
If this request fails, the server is not ODK Aggregate-based, or the
submission is not present -- ignore the error and proceed.

Proceed with the usual POST work flow, using the filtered list of
attachments to POST.

··· ------- Mitch

On Wed, Jul 2, 2014 at 8:12 PM, bobachgill@gmail.com wrote:

Just for grins...
From what I see here in China... when the export flood gate is opened...
smart phones in the world will all be stamped "made in China". They are
using their own population to test them... the $100 to $200 models looking
really good!

--
You received this message because you are subscribed to the Google Groups
"ODK Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to opendatakit-developers+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--
Mitch Sundt
Software Engineer
University of Washington
mitchellsundt@gmail.com

8

I am finding that as I am putting more stress on the ODK design than may have been originally envisioned... and therefore issues seem to be surfacing.

Is it helpful to the ODK dev community to discus in this forum why uploading a form with 1742 3-12MB media files is only getting half way through the send from a Youth hostel in Beijing?

9

(no problem; interesting to hear the extreme cases for the tool).

As for the transmission process, it has been a while since I've looked at
it; it was intended that a transmission failure would trigger one re-send
of that data, then advance to the next set of attachments. This may or may
not still be the case (ODK Collect tries to bucket attachments in 10MB
chunks for transmission to the server).

Attempting to re-submit these failed transmissions (i.e.., via the Send
Finalized Form screen) will always send all the data to the server all
over again -- the OpenRosa submission protocol does not have a mechanism
to find out what files the server already has and omit re-sending those.

··· ------------------------- The ability to successfully send media attachments depends upon the configuration of the server and the speed of your network from the device to the server.

First, on the server, the JVM configuration must be large enough to hold
the entire contents of your largest media attachment in memory. To be
conservative, I would recommend adding 100MB to that size for the minimum
size of your server JVM.

So if you are sending a 100MB file, you will need a JVM configured for
200MB.

Second, sending all of that data will take time. There are timeouts for
the communications channel between your device and the server that may
trigger if your network is slow. For example, on Google AppEngine, there is
an overall 60-second request timeout -- if the transmission and processing
takes longer than 60 seconds, Google will terminate the interaction in a
very harsh way.

To put this in perspective, if you have a 3Mb/second pathway to the server
(an unusually-fast 3G connection), then a 100MB file (800Mb), would take
~270 seconds to transmit -- and that is just sending it. The server cannot
process the file until it is fully received from the client, and its
processing entails sending the data to other servers for storage. Even at
the very-fast data rates within a computing cluster, this would take
several more seconds.

I suspect the network speed is what is killing you...

Mitch

On Wed, Jul 2, 2014 at 2:49 AM, Christopher Robert chrislrobert@gmail.com wrote:

Hi Yaw,

Yes, sorry for the trouble. We are working with Bob via our support
system. As he says, he is sending a form from China with over 1,500
multi-megabyte media attachments. This may be a case where sending via the
server is just not going to work as there may be numerous challenges. We're
working to support him, but his is obviously a challenging and very unusual
case. (He also gives us only a few hours to resolve something before coming
to this forum.)

Best,

Chris

On Tue, Jul 1, 2014 at 9:12 PM, Yaw Anokwa yanokwa@nafundi.com wrote:

Bob,

Sounds like you should email the guys at SurveyCTO...

Yaw

Need ODK services? http://nafundi.com provides form design, server
setup, professional support, and software development for ODK.

On Tue, Jul 1, 2014 at 6:10 PM, bobachgill@gmail.com wrote:

What would make the sending of a form with media files get stuck in a
loop trying over and over to get past sending that n'th media file in a
form?

What I have tried...

  1. closing and restarting SurveyCTO and restarting the send
  2. checked my connection --- I can browse the internet from the device
  3. waiting enough time --- my phone data usage shows that the app has
    sent 4x the data required to send the form
  4. make sure I am sending only one form

After all this only 650 of the media files have successfully send and
350 remain unsent.

The phone's data traffic monitor shows consistent increase in data
movement. So I can deduce that SurveyCTO is attempting to send the same
n'th file over and over again... but having to resend it because of some
retry logic.

One thought is that one media file is 20 percent larger than the others
and the time stamp has it about 650 down the list of media files to upload.
If the send goes by time stamp order that would put it right about the
time when the process is getting stuck in a loop. Could it be that that
12MB file size is getting stuck in a loop of trying to upload?

If so... can I test this by removing or replacing that file with one of
the same name but smaller file size and restart the send to see if that
corrects the looping? As long as replace the suspect media file with one
smaller but the same name will that disturb the marked completed form?

Many thanks!
Bob

--
You received this message because you are subscribed to the Google
Groups "ODK Developers" group.
To unsubscribe from this group and stop receiving emails from it, send
an email to opendatakit-developers+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups
"ODK Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to opendatakit-developers+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups
"ODK Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to opendatakit-developers+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--
Mitch Sundt
Software Engineer
University of Washington
mitchellsundt@gmail.com

10

And if you have many people submitting forms all at once (e.g., all at the
end of the day), then you would want a farm of servers, much larger JVMs,
and some restrictions on the load balancer to evenly distribute the
requests across the servers.

··· On Wed, Jul 2, 2014 at 9:34 AM, Mitch Sundt wrote:

(no problem; interesting to hear the extreme cases for the tool).

As for the transmission process, it has been a while since I've looked at
it; it was intended that a transmission failure would trigger one re-send
of that data, then advance to the next set of attachments. This may or may
not still be the case (ODK Collect tries to bucket attachments in 10MB
chunks for transmission to the server).

Attempting to re-submit these failed transmissions (i.e.., via the Send
Finalized Form screen) will always send all the data to the server all
over again -- the OpenRosa submission protocol does not have a mechanism
to find out what files the server already has and omit re-sending those.

The ability to successfully send media attachments depends upon the
configuration of the server and the speed of your network from the device
to the server.

First, on the server, the JVM configuration must be large enough to
hold the entire contents of your largest media attachment in memory. To be
conservative, I would recommend adding 100MB to that size for the minimum
size of your server JVM.

So if you are sending a 100MB file, you will need a JVM configured for
200MB.

Second, sending all of that data will take time. There are timeouts for
the communications channel between your device and the server that may
trigger if your network is slow. For example, on Google AppEngine, there is
an overall 60-second request timeout -- if the transmission and processing
takes longer than 60 seconds, Google will terminate the interaction in a
very harsh way.

To put this in perspective, if you have a 3Mb/second pathway to the server
(an unusually-fast 3G connection), then a 100MB file (800Mb), would take
~270 seconds to transmit -- and that is just sending it. The server cannot
process the file until it is fully received from the client, and its
processing entails sending the data to other servers for storage. Even at
the very-fast data rates within a computing cluster, this would take
several more seconds.

I suspect the network speed is what is killing you...

Mitch

On Wed, Jul 2, 2014 at 2:49 AM, Christopher Robert <chrislrobert@gmail.com wrote:

Hi Yaw,

Yes, sorry for the trouble. We are working with Bob via our support
system. As he says, he is sending a form from China with over 1,500
multi-megabyte media attachments. This may be a case where sending via the
server is just not going to work as there may be numerous challenges. We're
working to support him, but his is obviously a challenging and very unusual
case. (He also gives us only a few hours to resolve something before coming
to this forum.)

Best,

Chris

On Tue, Jul 1, 2014 at 9:12 PM, Yaw Anokwa yanokwa@nafundi.com wrote:

Bob,

Sounds like you should email the guys at SurveyCTO...

Yaw

Need ODK services? http://nafundi.com provides form design, server
setup, professional support, and software development for ODK.

On Tue, Jul 1, 2014 at 6:10 PM, bobachgill@gmail.com wrote:

What would make the sending of a form with media files get stuck in a
loop trying over and over to get past sending that n'th media file in a
form?

What I have tried...

  1. closing and restarting SurveyCTO and restarting the send
  2. checked my connection --- I can browse the internet from the device
  3. waiting enough time --- my phone data usage shows that the app has
    sent 4x the data required to send the form
  4. make sure I am sending only one form

After all this only 650 of the media files have successfully send and
350 remain unsent.

The phone's data traffic monitor shows consistent increase in data
movement. So I can deduce that SurveyCTO is attempting to send the same
n'th file over and over again... but having to resend it because of some
retry logic.

One thought is that one media file is 20 percent larger than the
others and the time stamp has it about 650 down the list of media files to
upload. If the send goes by time stamp order that would put it right
about the time when the process is getting stuck in a loop. Could it be
that that 12MB file size is getting stuck in a loop of trying to upload?

If so... can I test this by removing or replacing that file with one
of the same name but smaller file size and restart the send to see if that
corrects the looping? As long as replace the suspect media file with one
smaller but the same name will that disturb the marked completed form?

Many thanks!
Bob

--
You received this message because you are subscribed to the Google
Groups "ODK Developers" group.
To unsubscribe from this group and stop receiving emails from it, send
an email to opendatakit-developers+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google
Groups "ODK Developers" group.
To unsubscribe from this group and stop receiving emails from it, send
an email to opendatakit-developers+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups
"ODK Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to opendatakit-developers+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--
Mitch Sundt
Software Engineer
University of Washington
mitchellsundt@gmail.com

--
Mitch Sundt
Software Engineer
University of Washington
mitchellsundt@gmail.com

11

Thanks, Mitch.

Memory should be no trouble on our servers (we have multiple gigabytes of
free JVM heap space at any given time), and we don't have the GAE timeouts
to worry about (and in fact we have significantly boosted Tomcat's default
timeouts).

We've advised that the server be bypassed in this case since it seems
unlikely to be able to get all 1,500+ big attachments through in one go. It
seems like it gets a hiccup somewhere in the multi-hour upload process,
then starts again.

Thanks again,

Chris

··· On Wed, Jul 2, 2014 at 12:34 PM, Mitch Sundt wrote:

(no problem; interesting to hear the extreme cases for the tool).

As for the transmission process, it has been a while since I've looked at
it; it was intended that a transmission failure would trigger one re-send
of that data, then advance to the next set of attachments. This may or may
not still be the case (ODK Collect tries to bucket attachments in 10MB
chunks for transmission to the server).

Attempting to re-submit these failed transmissions (i.e.., via the Send
Finalized Form screen) will always send all the data to the server all
over again -- the OpenRosa submission protocol does not have a mechanism
to find out what files the server already has and omit re-sending those.

The ability to successfully send media attachments depends upon the
configuration of the server and the speed of your network from the device
to the server.

First, on the server, the JVM configuration must be large enough to
hold the entire contents of your largest media attachment in memory. To be
conservative, I would recommend adding 100MB to that size for the minimum
size of your server JVM.

So if you are sending a 100MB file, you will need a JVM configured for
200MB.

Second, sending all of that data will take time. There are timeouts for
the communications channel between your device and the server that may
trigger if your network is slow. For example, on Google AppEngine, there is
an overall 60-second request timeout -- if the transmission and processing
takes longer than 60 seconds, Google will terminate the interaction in a
very harsh way.

To put this in perspective, if you have a 3Mb/second pathway to the server
(an unusually-fast 3G connection), then a 100MB file (800Mb), would take
~270 seconds to transmit -- and that is just sending it. The server cannot
process the file until it is fully received from the client, and its
processing entails sending the data to other servers for storage. Even at
the very-fast data rates within a computing cluster, this would take
several more seconds.

I suspect the network speed is what is killing you...

Mitch

On Wed, Jul 2, 2014 at 2:49 AM, Christopher Robert <chrislrobert@gmail.com wrote:

Hi Yaw,

Yes, sorry for the trouble. We are working with Bob via our support
system. As he says, he is sending a form from China with over 1,500
multi-megabyte media attachments. This may be a case where sending via the
server is just not going to work as there may be numerous challenges. We're
working to support him, but his is obviously a challenging and very unusual
case. (He also gives us only a few hours to resolve something before coming
to this forum.)

Best,

Chris

On Tue, Jul 1, 2014 at 9:12 PM, Yaw Anokwa yanokwa@nafundi.com wrote:

Bob,

Sounds like you should email the guys at SurveyCTO...

Yaw

Need ODK services? http://nafundi.com provides form design, server
setup, professional support, and software development for ODK.

On Tue, Jul 1, 2014 at 6:10 PM, bobachgill@gmail.com wrote:

What would make the sending of a form with media files get stuck in a
loop trying over and over to get past sending that n'th media file in a
form?

What I have tried...

  1. closing and restarting SurveyCTO and restarting the send
  2. checked my connection --- I can browse the internet from the device
  3. waiting enough time --- my phone data usage shows that the app has
    sent 4x the data required to send the form
  4. make sure I am sending only one form

After all this only 650 of the media files have successfully send and
350 remain unsent.

The phone's data traffic monitor shows consistent increase in data
movement. So I can deduce that SurveyCTO is attempting to send the same
n'th file over and over again... but having to resend it because of some
retry logic.

One thought is that one media file is 20 percent larger than the
others and the time stamp has it about 650 down the list of media files to
upload. If the send goes by time stamp order that would put it right
about the time when the process is getting stuck in a loop. Could it be
that that 12MB file size is getting stuck in a loop of trying to upload?

If so... can I test this by removing or replacing that file with one
of the same name but smaller file size and restart the send to see if that
corrects the looping? As long as replace the suspect media file with one
smaller but the same name will that disturb the marked completed form?

Many thanks!
Bob

--
You received this message because you are subscribed to the Google
Groups "ODK Developers" group.
To unsubscribe from this group and stop receiving emails from it, send
an email to opendatakit-developers+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google
Groups "ODK Developers" group.
To unsubscribe from this group and stop receiving emails from it, send
an email to opendatakit-developers+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups
"ODK Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to opendatakit-developers+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--
Mitch Sundt
Software Engineer
University of Washington
mitchellsundt@gmail.com

--
You received this message because you are subscribed to the Google Groups
"ODK Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to opendatakit-developers+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

12

(no problem; interesting to hear the extreme cases for the tool).

As for the transmission process, it has been a while
since I've looked at it; it was intended that a transmission failure would trigger one re-send of that data, then advance to the next set of attachments. This may or may not still be the case (ODK Collect tries to bucket attachments in 10MB chunks for transmission to the server).

Attempting to re-submit these failed
transmissions (i.e.., via the Send Finalized Form screen) will always send all the data to the server all over again
-- the OpenRosa submission protocol does not have a mechanism to find
out what files the server already has and omit re-sending those.

The ability to successfully send media attachments depends upon the configuration of the server and the speed of your network from the device to the server.

So what would it take to check and compare the hash of media files on the Android side against the hashes on the server before the large media files are sent to the server so that restarts after hiccups only will take a few seconds to a couple minutes to check which files are already home on the server?

Given that media file capability was added to the ODK life cycle it seems like the ODK feature "Restart from where left off" capability is crippled if a restart involves more than just resending a bunch of text files??

As for the offline option... The offline approach for sneaker-netting the data back has it's own issues. If devices randomly choose to store the data on internal vs sd then the enumerator has to become a techie to figure out how to off load the data to sd in order to postal mail the sd. I have yet to find a phone here in China that saves to the SD. :frowning:

So which coding is easier to do??

  1. check hashes before send [to make restarts more tolerable]
    OR
  2. add data storage path to the config [to makes sure data is stored on sd]

Number 2 has other reasons why it would be good to do but Yaw says it is going to be difficult to do.

Number 1 is nice because then enumerators just hit "Send" with out the hassle of mailing in an SD

Thank you guys for all your help in thinking through this for me. You are the best!

Bob Achgill