[ODK Developers] Re: Sending stuck in a loop

Development
1

Since the Android device is the origination point of files being sent up to the server and the desire it to avoid unnecessarily resending the same large media files to the server... would it work just to simply rename files at they are successfully sent to the server? e.g. ########LargeMediaAlreadySent.mp4 This way the renamed media file(s) won't be found to be sent upon re initiation of the send. To my knowledge I don't know that files already sent will ever be used again in the life cycle of a submitted form. Aren't already submitted forms just hanging around till they are deleted? Renaming is better than just programmatically deleting the sent files in case you later need to debug why a form is having trouble being sent. "Ahhah, it appears that all the files got sent!" "Or, all but this one got renamed so ... let's focus on why it did not get sent."

For example: In driving from LA to DC and the GPS has a hiccup while in a rain storm... you would not need to go all the way back to LA and start over... just recalculate from the latest known position. In this case renaming files as you successfully upload each one determines that they will not be found to be sent again as the send recalculates what is left to send. This might drive your error message crazy if their is an error message for File not found... but that is easy to address by adding a condition to check if the file was already renamed because it was already sent.

Would this proposed fix mess with the Open Rosa standard? At simplest it is just a rename line of code and a few more lines to add the case to file not found error message processing.

What do you think?

Original message

··· From: "Mitch Sundt" To: yanokwa@nafundi.com; crobert@surveycto.com; opendatakit-developers@googlegroups.com; Dated: 7/3/2014 11:11:27 AM Subject: Re: [ODK Developers] Re: Sending stuck in a loop

Long term, this would be a useful discussion to open up with all the OpenRosa server implementers -- adding a mechanism to the protocol to obtain the list of media attachments already uploaded to the server, along with their md5 hashes.

Short term, if you are communicating with an ODK Aggregate 1.x-derived server, the code change to ODK Collect would be:

Obtain list of media attachment files for the submission from the device SD card / external storage location.

Issue Head request to establish authentication / authorization for session.

If attempting to re-send a submission that previously failed, issue an ODK Briefcase http://code.google.com/p/opendatakit/wiki/BriefcaseAggregateAPI GET view/downloadSubmission request to the server.

If this is successful, filter the list of media attachments to exclude those that the server reports as having been successfully uploaded.

If this request fails, the server is not ODK Aggregate-based, or the submission is not present -- ignore the error and proceed.

Proceed with the usual POST work flow, using the filtered list of attachments to POST.

Mitch

On Wed, Jul 2, 2014 at 8:12 PM, bobachgill@gmail.com wrote:

Just for grins...
From what I see here in China... when the export flood gate is opened... smart phones in the world will all be stamped "made in China". They are using their own population to test them... the $100 to $200 models looking really good!

--
You received this message because you are subscribed to the Google Groups "ODK Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to opendatakit-developers+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--
Mitch Sundt
Software Engineer
University of Washington
mitchellsundt@gmail.com

2

Bob,

If you are looking for a something quick that would work for your app,
this sounds like it'd work. You probably couldn't use Briefcase (or
other downstream tools) to pull from Collect and you'd probably have
to change form deletion so it removes the file as well.

I wouldn't call it a fix though. The fix is what Mitch described
earlier. And said fix also has the huge benefit of being useful to the
entire community.

Yaw

··· -- Need ODK services? http://nafundi.com provides form design, server setup, professional support, and software development for ODK. On Fri, Jul 11, 2014 at 2:42 PM, Bob Achgill wrote: > Since the Android device is the origination point of files being sent up to > the server and the desire it to avoid unnecessarily resending the same large > media files to the server... would it work just to simply rename files at > they are successfully sent to the server? e.g. > ########LargeMediaAlreadySent.mp4 This way the renamed media file(s) won't > be found to be sent upon re initiation of the send. To my knowledge I don't > know that files already sent will ever be used again in the life cycle of a > submitted form. Aren't already submitted forms just hanging around till > they are deleted? Renaming is better than just programmatically deleting > the sent files in case you later need to debug why a form is having trouble > being sent. "Ahhah, it appears that all the files got sent!" "Or, all but > this one got renamed so ... let's focus on why it did not get sent." > > For example: In driving from LA to DC and the GPS has a hiccup while in a > rain storm... you would not need to go all the way back to LA and start > over... just recalculate from the latest known position. In this case > renaming files as you successfully upload each one determines that they will > not be found to be sent again as the send recalculates what is left to send. > This might drive your error message crazy if their is an error message for > File not found... but that is easy to address by adding a condition to check > if the file was already renamed because it was already sent. > > Would this proposed fix mess with the Open Rosa standard? At simplest it is > just a rename line of code and a few more lines to add the case to file not > found error message processing. > > What do you think? > > > ________________________________ > Original message > From: "Mitch Sundt" > To: yanokwa@nafundi.com; crobert@surveycto.com; > opendatakit-developers@googlegroups.com; > Dated: 7/3/2014 11:11:27 AM > Subject: Re: [ODK Developers] Re: Sending stuck in a loop > > Long term, this would be a useful discussion to open up with all the > OpenRosa server implementers -- adding a mechanism to the protocol to obtain > the list of media attachments already uploaded to the server, along with > their md5 hashes. > > Short term, if you are communicating with an ODK Aggregate 1.x-derived > server, the code change to ODK Collect would be: > > Obtain list of media attachment files for the submission from the device SD > card / external storage location. > > Issue Head request to establish authentication / authorization for session. > > If attempting to re-send a submission that previously failed, issue an ODK > Briefcase http://code.google.com/p/opendatakit/wiki/BriefcaseAggregateAPI > GET view/downloadSubmission request to the server. > > If this is successful, filter the list of media attachments to exclude those > that the server reports as having been successfully uploaded. > If this request fails, the server is not ODK Aggregate-based, or the > submission is not present -- ignore the error and proceed. > > Proceed with the usual POST work flow, using the filtered list of > attachments to POST. > > ------- > Mitch > > > On Wed, Jul 2, 2014 at 8:12 PM, wrote: >> >> Just for grins... >> From what I see here in China... when the export flood gate is opened... >> smart phones in the world will all be stamped "made in China". They are >> using their own population to test them... the $100 to $200 models looking >> really good! >> >> -- >> You received this message because you are subscribed to the Google Groups >> "ODK Developers" group. >> To unsubscribe from this group and stop receiving emails from it, send an >> email to opendatakit-developers+unsubscribe@googlegroups.com. >> For more options, visit https://groups.google.com/d/optout. > > > > > -- > Mitch Sundt > Software Engineer > University of Washington > mitchellsundt@gmail.com