Hi everybody,
is there any procedure that allow the users (Collectors) to download only
csv files from Aggregate.... instead of downloading both forms and files
if they download also the form the list of available forms soon will be
filled. The form didn't changed at all. The only think that changed is the
csv file.
Thanks
Panos
Hi Panos,
Don't think so, but might be pretty easy to add. In the download
activity, you could check the MD5 hashes of the files you have with
the files on the server and download the stuff that has changed. I
don't remember if the manifest that ODK servers spit out have hashes
of multimedia, so start there.
Yaw
On Fri, May 30, 2014 at 5:00 AM, Panos Papadatos ppapadatosgr@gmail.com wrote:
Hi everybody,
is there any procedure that allow the users (Collectors) to download only
csv files from Aggregate.... instead of downloading both forms and filesif they download also the form the list of available forms soon will be
filled. The form didn't changed at all. The only think that changed is the
csv file.Thanks
Panos--
You received this message because you are subscribed to the Google Groups
"ODK Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to opendatakit-developers+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
Chris & Meletis: there were some changes to the form download logic that
broke the efficiency of the download.
Previously, during the download, the md5 hash of the local file would be
compared against the manifest, and the file would only be downloaded if the
hash differed. With the use of a temp directory in the cache folder, you
need to change the code to reference back to the original media folder and
copy the file from there rather than take a network traffic hit. Right
now, every time you get a form from the server, it pulls everything down.
The server reports the hash value it has for the XForms file in the list of
available forms: https://opendatakit.appspot.com/xformsList
Panos:
pre-1.4.3, the download would only download the media files that had
changed. I don't know whether it ever optimized on the form definition
change. You can certainly skip the downloadXForm() call if the md5 hash is
unchanged.
On Fri, May 30, 2014 at 7:54 AM, Yaw Anokwa yanokwa@nafundi.com wrote:
Hi Panos,
Don't think so, but might be pretty easy to add. In the download
activity, you could check the MD5 hashes of the files you have with
the files on the server and download the stuff that has changed. I
don't remember if the manifest that ODK servers spit out have hashes
of multimedia, so start there.Yaw
Need ODK services? http://nafundi.com provides form design, server
setup, professional support, and software development for ODK.On Fri, May 30, 2014 at 5:00 AM, Panos Papadatos ppapadatosgr@gmail.com wrote:
Hi everybody,
is there any procedure that allow the users (Collectors) to download only
csv files from Aggregate.... instead of downloading both forms and filesif they download also the form the list of available forms soon will be
filled. The form didn't changed at all. The only think that changed is
the
csv file.Thanks
Panos--
You received this message because you are subscribed to the Google Groups
"ODK Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to opendatakit-developers+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.--
You received this message because you are subscribed to the Google Groups
"ODK Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to opendatakit-developers+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
--
Mitch Sundt
Software Engineer
University of Washington
mitchellsundt@gmail.com
Mitch,
Are you talking about when Collect re-downloads a form? So when you go to
Get Blank Form and re-download a form you already have?
Thanks,
Chris
Chris & Meletis: there were some changes to the form download logic that
broke the efficiency of the download.Previously, during the download, the md5 hash of the local file would be
compared against the manifest, and the file would only be downloaded if the
hash differed. With the use of a temp directory in the cache folder, you
need to change the code to reference back to the original media folder and
copy the file from there rather than take a network traffic hit. Right
now, every time you get a form from the server, it pulls everything down.The server reports the hash value it has for the XForms file in the list
of available forms: https://opendatakit.appspot.com/xformsListPanos:
pre-1.4.3, the download would only download the media files that had
changed. I don't know whether it ever optimized on the form definition
change. You can certainly skip the downloadXForm() call if the md5 hash is
unchanged.
Mitch
On Fri, May 30, 2014 at 7:54 AM, Yaw Anokwa yanokwa@nafundi.com wrote:
Hi Panos,
Don't think so, but might be pretty easy to add. In the download
activity, you could check the MD5 hashes of the files you have with
the files on the server and download the stuff that has changed. I
don't remember if the manifest that ODK servers spit out have hashes
of multimedia, so start there.Yaw
Need ODK services? http://nafundi.com provides form design, server
setup, professional support, and software development for ODK.On Fri, May 30, 2014 at 5:00 AM, Panos Papadatos ppapadatosgr@gmail.com wrote:
Hi everybody,
is there any procedure that allow the users (Collectors) to download
only
csv files from Aggregate.... instead of downloading both forms and filesif they download also the form the list of available forms soon will be
filled. The form didn't changed at all. The only think that changed is
the
csv file.Thanks
Panos--
You received this message because you are subscribed to the Google
Groups
"ODK Developers" group.
To unsubscribe from this group and stop receiving emails from it, send
an
email to opendatakit-developers+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.--
You received this message because you are subscribed to the Google Groups
"ODK Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to opendatakit-developers+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.--
Mitch Sundt
Software Engineer
University of Washington
mitchellsundt@gmail.com--
You received this message because you are subscribed to the Google Groups
"ODK Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to opendatakit-developers+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
Mitch,
Yes, that was our fault -- sorry. When we made the downloads "all or
nothing" for robustness, that issue came in for subsequent downloads of a
form you already have. Meletis has pushed a fix:
For what it's worth, Meletis points out that, while the server delivers a
hash for the .xml file as well, that's ignored by Collect and it always
re-downloads that. That seems to be a long-standing issue.
(As for the relationship of any of this to Panos's issue, I'm not sure..!
Sorry, Panos.)
Best,
Chris
Chris & Meletis: there were some changes to the form download logic that
broke the efficiency of the download.Previously, during the download, the md5 hash of the local file would be
compared against the manifest, and the file would only be downloaded if the
hash differed. With the use of a temp directory in the cache folder, you
need to change the code to reference back to the original media folder and
copy the file from there rather than take a network traffic hit. Right
now, every time you get a form from the server, it pulls everything down.The server reports the hash value it has for the XForms file in the list
of available forms: https://opendatakit.appspot.com/xformsListPanos:
pre-1.4.3, the download would only download the media files that had
changed. I don't know whether it ever optimized on the form definition
change. You can certainly skip the downloadXForm() call if the md5 hash is
unchanged.
Mitch
On Fri, May 30, 2014 at 7:54 AM, Yaw Anokwa yanokwa@nafundi.com wrote:
Hi Panos,
Don't think so, but might be pretty easy to add. In the download
activity, you could check the MD5 hashes of the files you have with
the files on the server and download the stuff that has changed. I
don't remember if the manifest that ODK servers spit out have hashes
of multimedia, so start there.Yaw
Need ODK services? http://nafundi.com provides form design, server
setup, professional support, and software development for ODK.On Fri, May 30, 2014 at 5:00 AM, Panos Papadatos ppapadatosgr@gmail.com wrote:
Hi everybody,
is there any procedure that allow the users (Collectors) to download
only
csv files from Aggregate.... instead of downloading both forms and filesif they download also the form the list of available forms soon will be
filled. The form didn't changed at all. The only think that changed is
the
csv file.Thanks
Panos--
You received this message because you are subscribed to the Google
Groups
"ODK Developers" group.
To unsubscribe from this group and stop receiving emails from it, send
an
email to opendatakit-developers+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.--
You received this message because you are subscribed to the Google Groups
"ODK Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to opendatakit-developers+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.--
Mitch Sundt
Software Engineer
University of Washington
mitchellsundt@gmail.com--
You received this message because you are subscribed to the Google Groups
"ODK Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to opendatakit-developers+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
Yes of cource you have right, but every time you download even the same form the form list will be expanded...
Thanks for your reply
Panos
Thanks Chris & Meletis for the fix.
Yes, the check of the form was never present.
Panos:
If you have a proposed improvement, open an issue with the patch, and I
will take a look at it. I don't understand what you see as an issue -- the
list of forms, and the manifest for a form will ALWAYS need to be
downloaded. That is the only way ODK Collect knows what files are on the
server. If you are using media files or large CSVs for external datasets,
the size of the manifest should be very small relative to the sizes of the
files themselves, so downloading the manifest should not be a significant
fraction of your bandwidth. The server has no API to ask for just the hash
of one file. And it is unclear how this would mesh with the robustness
changes SurveyCTO contributed, which ensure that any sync from the server
is a full file sync -- partial syncs are explicitly prevented. It sounds
like what you want is to re-introduce partial sync states, which, I think,
would be a bad idea.
Mitch
Mitch,
Yes, that was our fault -- sorry. When we made the downloads "all or
nothing" for robustness, that issue came in for subsequent downloads of a
form you already have. Meletis has pushed a fix:For what it's worth, Meletis points out that, while the server delivers a
hash for the .xml file as well, that's ignored by Collect and it always
re-downloads that. That seems to be a long-standing issue.(As for the relationship of any of this to Panos's issue, I'm not sure..!
Sorry, Panos.)Best,
Chris
On Sat, May 31, 2014 at 12:59 AM, Mitch Sundt mitchellsundt@gmail.com wrote:
Chris & Meletis: there were some changes to the form download logic that
broke the efficiency of the download.Previously, during the download, the md5 hash of the local file would be
compared against the manifest, and the file would only be downloaded if the
hash differed. With the use of a temp directory in the cache folder, you
need to change the code to reference back to the original media folder and
copy the file from there rather than take a network traffic hit. Right
now, every time you get a form from the server, it pulls everything down.The server reports the hash value it has for the XForms file in the list
of available forms: https://opendatakit.appspot.com/xformsListPanos:
pre-1.4.3, the download would only download the media files that had
changed. I don't know whether it ever optimized on the form definition
change. You can certainly skip the downloadXForm() call if the md5 hash is
unchanged.
Mitch
On Fri, May 30, 2014 at 7:54 AM, Yaw Anokwa yanokwa@nafundi.com wrote:
Hi Panos,
Don't think so, but might be pretty easy to add. In the download
activity, you could check the MD5 hashes of the files you have with
the files on the server and download the stuff that has changed. I
don't remember if the manifest that ODK servers spit out have hashes
of multimedia, so start there.Yaw
Need ODK services? http://nafundi.com provides form design, server
setup, professional support, and software development for ODK.On Fri, May 30, 2014 at 5:00 AM, Panos Papadatos ppapadatosgr@gmail.com wrote:
Hi everybody,
is there any procedure that allow the users (Collectors) to download
only
csv files from Aggregate.... instead of downloading both forms and
filesif they download also the form the list of available forms soon will be
filled. The form didn't changed at all. The only think that changed is
the
csv file.Thanks
Panos--
You received this message because you are subscribed to the Google
Groups
"ODK Developers" group.
To unsubscribe from this group and stop receiving emails from it, send
an
email to opendatakit-developers+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.--
You received this message because you are subscribed to the Google
Groups "ODK Developers" group.
To unsubscribe from this group and stop receiving emails from it, send
an email to opendatakit-developers+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.--
Mitch Sundt
Software Engineer
University of Washington
mitchellsundt@gmail.com--
You received this message because you are subscribed to the Google Groups
"ODK Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to opendatakit-developers+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.--
You received this message because you are subscribed to the Google Groups
"ODK Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to opendatakit-developers+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
--
Mitch Sundt
Software Engineer
University of Washington
mitchellsundt@gmail.com
Yes I understand the procedure...
It was my fault because i thought that xml form and media files are two
parts but in fact xml form and media files are one part in mysql and not in
file system.
The question was if it was possible to download only changed media files
instead of xml form and media files (partial sync), because media (csv)
files will change every day... It's not static data..
But as i understand this is not possible .
Anyway thanks for your reply.
Panos
To be clear, when you download a form from the server, ODK Collect only
downloads the individual files that have changed or those that were not
present when the form was last downloaded.
The latest release (1.4.3) ensures that all the files that are on the
server are downloaded together, and only once they are all successfully
transferred, does it swap ODK Collect to reference this newly downloaded
form. It the process of making this robustness improvement, the path that
optimized the downloads to avoid downloading the unchanging files was
broken; this will be fixed in the next release.
The fact that the files are stored as blobs in the datastore is an
independent issue. They are all tracked and changed independently.
Mitch
Yes I understand the procedure...
It was my fault because i thought that xml form and media files are two
parts but in fact xml form and media files are one part in mysql and not in
file system.The question was if it was possible to download only changed media files
instead of xml form and media files (partial sync), because media (csv)
files will change every day... It's not static data..But as i understand this is not possible .
Anyway thanks for your reply.
PanosOn Monday, June 2, 2014 8:02:02 PM UTC+3, Mitch wrote:
Thanks Chris & Meletis for the fix.
Yes, the check of the form was never present.
Panos:
If you have a proposed improvement, open an issue with the patch, and I
will take a look at it. I don't understand what you see as an issue -- the
list of forms, and the manifest for a form will ALWAYS need to be
downloaded. That is the only way ODK Collect knows what files are on the
server. If you are using media files or large CSVs for external datasets,
the size of the manifest should be very small relative to the sizes of the
files themselves, so downloading the manifest should not be a significant
fraction of your bandwidth. The server has no API to ask for just the hash
of one file. And it is unclear how this would mesh with the robustness
changes SurveyCTO contributed, which ensure that any sync from the server
is a full file sync -- partial syncs are explicitly prevented. It sounds
like what you want is to re-introduce partial sync states, which, I think,
would be a bad idea.Mitch
On Sun, Jun 1, 2014 at 8:04 AM, Christopher Robert cro...@surveycto.com wrote:
Mitch,
Yes, that was our fault -- sorry. When we made the downloads "all or
nothing" for robustness, that issue came in for subsequent downloads of a
form you already have. Meletis has pushed a fix:https://code.google.com/p/opendatakit/source/detail?r=
d19f1f9c2393a95440927547aa2c6c695304ab0b&name=surveycto&repo=collectFor what it's worth, Meletis points out that, while the server delivers
a hash for the .xml file as well, that's ignored by Collect and it always
re-downloads that. That seems to be a long-standing issue.(As for the relationship of any of this to Panos's issue, I'm not
sure..! Sorry, Panos.)Best,
Chris
On Sat, May 31, 2014 at 12:59 AM, Mitch Sundt mitche...@gmail.com wrote:
Chris & Meletis: there were some changes to the form download logic
that broke the efficiency of the download.Previously, during the download, the md5 hash of the local file would
be compared against the manifest, and the file would only be downloaded if
the hash differed. With the use of a temp directory in the cache folder,
you need to change the code to reference back to the original media folder
and copy the file from there rather than take a network traffic hit. Right
now, every time you get a form from the server, it pulls everything down.The server reports the hash value it has for the XForms file in the
list of available forms: https://opendatakit.appspot.com/xformsListPanos:
pre-1.4.3, the download would only download the media files that had
changed. I don't know whether it ever optimized on the form definition
change. You can certainly skip the downloadXForm() call if the md5 hash is
unchanged.
Mitch
On Fri, May 30, 2014 at 7:54 AM, Yaw Anokwa yan...@nafundi.com wrote:
Hi Panos,
Don't think so, but might be pretty easy to add. In the download
activity, you could check the MD5 hashes of the files you have with
the files on the server and download the stuff that has changed. I
don't remember if the manifest that ODK servers spit out have hashes
of multimedia, so start there.Yaw
Need ODK services? http://nafundi.com provides form design, server
setup, professional support, and software development for ODK.On Fri, May 30, 2014 at 5:00 AM, Panos Papadatos ppapad...@gmail.com wrote:
Hi everybody,
is there any procedure that allow the users (Collectors) to download
only
csv files from Aggregate.... instead of downloading both forms and
filesif they download also the form the list of available forms soon will
be
filled. The form didn't changed at all. The only think that changed
is the
csv file.Thanks
Panos--
You received this message because you are subscribed to the Google
Groups
"ODK Developers" group.
To unsubscribe from this group and stop receiving emails from it,
send an
email to opendatakit-developers+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.--
You received this message because you are subscribed to the Google
Groups "ODK Developers" group.
To unsubscribe from this group and stop receiving emails from it, send
an email to opendatakit-developers+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.--
Mitch Sundt
Software Engineer
University of Washington
mitche...@gmail.com--
You received this message because you are subscribed to the Google
Groups "ODK Developers" group.
To unsubscribe from this group and stop receiving emails from it, send
an email to opendatakit-developers+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.--
You received this message because you are subscribed to the Google
Groups "ODK Developers" group.
To unsubscribe from this group and stop receiving emails from it, send
an email to opendatakit-developers+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.--
Mitch Sundt
Software Engineer
University of Washington
mitche...@gmail.com--
You received this message because you are subscribed to the Google Groups
"ODK Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to opendatakit-developers+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
--
Mitch Sundt
Software Engineer
University of Washington
mitchellsundt@gmail.com