Briefcase "Pull" function takes longer with each new PDA we pull data from

In pilot tests the past few weeks, my team has noticed that the Briefcase
"Pull" function takes longer with each new PDA we pull data from. The first
time we pull data from the ODK folder on a PDA, it takes just a few seconds
for the data to be downloaded. By the 10th PDA it takes over 10 minutes.

This could certainly be bc we have a very long survey, and we accept that
we will face such hurdles. But if there's a way to minimize the time it
takes to pull data with Briefcase, please let us know.

Thanks,
Sam

I believe this is due to an omission of the 'delete saved form' step in
your device handling procedures...

When using ODK Briefcase, it is important to understand that it does not
alter the contents of any 'Pull' source, including the ODK Collect device's
SDCard.

Therefore, after you 'Pull' data off the device, it is still on that device
-- you must then manually delete the data from the device (via ODK
Collect - Delete Saved Form, Saved Forms screen). We do it this way
because we don't want to be responsible for destroying your data, and
different groups may follow different procedures or have different process
requirements (e.g., pull off the device with Briefcase and also send to ODK
Aggregate).

If you don't delete these files, on a subsequent 'Pull', because ODK
Briefcase maintains no record of what it previously pulled off of a device
and it will end up pulling the already-pulled data (again), in addition to
all the newly-collected data.

And here is where the growth in time occurs: under normal conditions, ODK
Briefcase will simply copy the data directory off the phone, and place it
into the ODK Briefcase storage location. This usually works; ODK Collect
names submission directories by their form name plus a creation timestamp
of the submission: 2013-03-07_16-21-10 (down to the second). It is
extremely unlikely that two surveyors would 'Fill Blank Form' at exactly
the same second, so name collisions are rare, but not impossible.

In the exceptional case that directory names do collide, ODK Briefcase then
starts adding _1, _2, .etc to the end of the directory name, looking for
the first not-yet-used name.

This last step is where your 10+ minute load occurs -- after cycling
through and not deleting already-pulled forms from your phone, ODK
Briefcase gets clogged with duplicates of all that data, and the
test-and-increment loop takes forever to complete.

Mitch

··· On Fri, Mar 8, 2013 at 9:33 AM, Sam wrote:

In pilot tests the past few weeks, my team has noticed that the Briefcase
"Pull" function takes longer with each new PDA we pull data from. The first
time we pull data from the ODK folder on a PDA, it takes just a few seconds
for the data to be downloaded. By the 10th PDA it takes over 10 minutes.

This could certainly be bc we have a very long survey, and we accept that
we will face such hurdles. But if there's a way to minimize the time it
takes to pull data with Briefcase, please let us know.

Thanks,
Sam

--

Post: opendatakit@googlegroups.com
Unsubscribe: opendatakit+unsubscribe@googlegroups.com
Options: http://groups.google.com/group/opendatakit?hl=en


You received this message because you are subscribed to the Google Groups
"ODK Community" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to opendatakit+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

--
Mitch Sundt
Software Engineer
University of Washington
mitchellsundt@gmail.com

Thanks Mitch, we might then begin to delete the saved forms on the PDAs.

However, this is happening now even with new surveys that we are loading
for the first time. The second PDA I tried to pull data from over 10
minutes ago says "SUCCESS!" under "Pull Status," but keeps showing
"Downloading.." and not allow me to switch to the Push or Export tabs.

Could this be another issue?

Thanks,
Sam

··· On Friday, March 8, 2013, Mitch Sundt wrote:

I believe this is due to an omission of the 'delete saved form' step in
your device handling procedures...

When using ODK Briefcase, it is important to understand that it does not
alter the contents of any 'Pull' source, including the ODK Collect device's
SDCard.

Therefore, after you 'Pull' data off the device, it is still on that
device -- you must then manually delete the data from the device (via
ODK Collect - Delete Saved Form, Saved Forms screen). We do it this way
because we don't want to be responsible for destroying your data, and
different groups may follow different procedures or have different process
requirements (e.g., pull off the device with Briefcase and also send to ODK
Aggregate).

If you don't delete these files, on a subsequent 'Pull', because ODK
Briefcase maintains no record of what it previously pulled off of a device
and it will end up pulling the already-pulled data (again), in addition to
all the newly-collected data.

And here is where the growth in time occurs: under normal conditions, ODK
Briefcase will simply copy the data directory off the phone, and place it
into the ODK Briefcase storage location. This usually works; ODK Collect
names submission directories by their form name plus a creation timestamp
of the submission: 2013-03-07_16-21-10 (down to the second). It is
extremely unlikely that two surveyors would 'Fill Blank Form' at exactly
the same second, so name collisions are rare, but not impossible.

In the exceptional case that directory names do collide, ODK Briefcase
then starts adding _1, _2, .etc to the end of the directory name, looking
for the first not-yet-used name.

This last step is where your 10+ minute load occurs -- after cycling
through and not deleting already-pulled forms from your phone, ODK
Briefcase gets clogged with duplicates of all that data, and the
test-and-increment loop takes forever to complete.

Mitch

On Fri, Mar 8, 2013 at 9:33 AM, Sam <samantha.elghanayan@gmail.com<javascript:_e({}, 'cvml', 'samantha.elghanayan@gmail.com');> wrote:

In pilot tests the past few weeks, my team has noticed that the Briefcase
"Pull" function takes longer with each new PDA we pull data from. The first
time we pull data from the ODK folder on a PDA, it takes just a few seconds
for the data to be downloaded. By the 10th PDA it takes over 10 minutes.

This could certainly be bc we have a very long survey, and we accept that
we will face such hurdles. But if there's a way to minimize the time it
takes to pull data with Briefcase, please let us know.

Thanks,
Sam

--

Post: opendatakit@googlegroups.com <javascript:_e({}, 'cvml',
'opendatakit@googlegroups.com');>
Unsubscribe: opendatakit+unsubscribe@googlegroups.com <javascript:_e({},
'cvml', 'opendatakit%2Bunsubscribe@googlegroups.com');>
Options: http://groups.google.com/group/opendatakit?hl=en


You received this message because you are subscribed to the Google Groups
"ODK Community" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to opendatakit+unsubscribe@googlegroups.com <javascript:_e({},
'cvml', 'opendatakit%2Bunsubscribe@googlegroups.com');>.
For more options, visit https://groups.google.com/groups/opt_out.

--
Mitch Sundt
Software Engineer
University of Washington
mitchellsundt@gmail.com <javascript:_e({}, 'cvml',
'mitchellsundt@gmail.com');>

--

Post: opendatakit@googlegroups.com <javascript:_e({}, 'cvml',
'opendatakit@googlegroups.com');>
Unsubscribe: opendatakit+unsubscribe@googlegroups.com <javascript:_e({},
'cvml', 'opendatakit%2Bunsubscribe@googlegroups.com');>
Options: http://groups.google.com/group/opendatakit?hl=en


You received this message because you are subscribed to a topic in the
Google Groups "ODK Community" group.
To unsubscribe from this topic, visit
https://groups.google.com/d/topic/opendatakit/d-h4CY5z8Aw/unsubscribe?hl=en
.
To unsubscribe from this group and all its topics, send an email to
opendatakit+unsubscribe@googlegroups.com <javascript:_e({}, 'cvml',
'opendatakit%2Bunsubscribe@googlegroups.com');>.
For more options, visit https://groups.google.com/groups/opt_out.

I just closed and reopened Briefcase and now the data loaded in about a
minute. We've noticed that it generally helps to restart Briefcase when
pulling takes too long. Would it still be linked to the same issue you
noted on your first post?
Thanks

··· On Sunday, March 10, 2013, Samantha Elghanayan wrote:

Thanks Mitch, we might then begin to delete the saved forms on the PDAs.

However, this is happening now even with new surveys that we are loading
for the first time. The second PDA I tried to pull data from over 10
minutes ago says "SUCCESS!" under "Pull Status," but keeps showing
"Downloading.." and not allow me to switch to the Push or Export tabs.

Could this be another issue?

Thanks,
Sam

On Friday, March 8, 2013, Mitch Sundt wrote:

I believe this is due to an omission of the 'delete saved form' step in
your device handling procedures...

When using ODK Briefcase, it is important to understand that it does not
alter the contents of any 'Pull' source, including the ODK Collect device's
SDCard.

Therefore, after you 'Pull' data off the device, it is still on that
device -- you must then manually delete the data from the device (via
ODK Collect - Delete Saved Form, Saved Forms screen). We do it this way
because we don't want to be responsible for destroying your data, and
different groups may follow different procedures or have different process
requirements (e.g., pull off the device with Briefcase and also send to ODK
Aggregate).

If you don't delete these files, on a subsequent 'Pull', because ODK
Briefcase maintains no record of what it previously pulled off of a device
and it will end up pulling the already-pulled data (again), in addition to
all the newly-collected data.

And here is where the growth in time occurs: under normal conditions, ODK
Briefcase will simply copy the data directory off the phone, and place it
into the ODK Briefcase storage location. This usually works; ODK Collect
names submission directories by their form name plus a creation timestamp
of the submission: 2013-03-07_16-21-10 (down to the second). It is
extremely unlikely that two surveyors would 'Fill Blank Form' at exactly
the same second, so name collisions are rare, but not impossible.

In the exceptional case that directory names do collide, ODK Briefcase
then starts adding _1, _2, .etc to the end of the directory name, looking
for the first not-yet-used name.

This last step is where your 10+ minute load occurs -- after cycling
through and not deleting already-pulled forms from your phone, ODK
Briefcase gets clogged with duplicates of all that data, and the
test-and-increment loop takes forever to complete.

Mitch

On Fri, Mar 8, 2013 at 9:33 AM, Sam samantha.elghanayan@gmail.comwrote:

In pilot tests the past few weeks, my team has noticed that
the Briefcase "Pull" function takes longer with each new PDA we pull data
from. The first time we pull data from the ODK folder on a PDA, it takes
just a few seconds for the data to be downloaded. By the 10th PDA it takes
over 10 minutes.

This could certainly be bc we have a very long survey, and we accept
that we will face such hurdles. But if there's a way to minimize the time
it takes to pull data with Briefcase, please let us know.

Thanks,
Sam

--

Post: opendatakit@googlegroups.com
Unsubscribe: opendatakit+unsubscribe@googlegroups.com
Options: http://groups.google.com/group/opendatakit?hl=en


You received this message because you are subscribed to the Google
Groups "ODK Community" group.
To unsubscribe from this group and stop receiving emails from it, send
an email to opendatakit+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

--
Mitch Sundt
Software Engineer
University of Washington
mitchellsundt@gmail.com

--

Post: opendatakit@googlegroups.com
Unsubscribe: opendatakit+unsubscribe@googlegroups.com
Options: http://groups.google.com/group/opendatakit?hl=en


You received this message because you are subscribed to a topic in the
Google Groups "ODK Community" group.
To unsubscribe from this topic, visit
https://groups.google.com/d/topic/opendatakit/d-h4CY5z8Aw/unsubscribe?hl=en
.
To unsubscribe from this group and all its topics, send an email to
opendatakit+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

Now that sounds like a bug.

Have you noticed if this happens consistently after the 2nd device is being
copied from, after the 3rd, etc?

And did anything fail on the earlier pulls? Or not complete with success?

If you can characterize the steps required to cause the problem, that would
be useful.

Mitch

··· On Sun, Mar 10, 2013 at 6:55 AM, Samantha Elghanayan < samantha.elghanayan@gmail.com> wrote:

I just closed and reopened Briefcase and now the data loaded in about a
minute. We've noticed that it generally helps to restart Briefcase when
pulling takes too long. Would it still be linked to the same issue you
noted on your first post?
Thanks

On Sunday, March 10, 2013, Samantha Elghanayan wrote:

Thanks Mitch, we might then begin to delete the saved forms on the PDAs.

However, this is happening now even with new surveys that we are loading
for the first time. The second PDA I tried to pull data from over 10
minutes ago says "SUCCESS!" under "Pull Status," but keeps showing
"Downloading.." and not allow me to switch to the Push or Export tabs.

Could this be another issue?

Thanks,
Sam

On Friday, March 8, 2013, Mitch Sundt wrote:

I believe this is due to an omission of the 'delete saved form' step in
your device handling procedures...

When using ODK Briefcase, it is important to understand that it does not
alter the contents of any 'Pull' source, including the ODK Collect device's
SDCard.

Therefore, after you 'Pull' data off the device, it is still on that
device -- you must then manually delete the data from the device (via
ODK Collect - Delete Saved Form, Saved Forms screen). We do it this way
because we don't want to be responsible for destroying your data, and
different groups may follow different procedures or have different process
requirements (e.g., pull off the device with Briefcase and also send to ODK
Aggregate).

If you don't delete these files, on a subsequent 'Pull', because ODK
Briefcase maintains no record of what it previously pulled off of a device
and it will end up pulling the already-pulled data (again), in addition to
all the newly-collected data.

And here is where the growth in time occurs: under normal conditions,
ODK Briefcase will simply copy the data directory off the phone, and place
it into the ODK Briefcase storage location. This usually works; ODK Collect
names submission directories by their form name plus a creation timestamp
of the submission: 2013-03-07_16-21-10 (down to the second). It is
extremely unlikely that two surveyors would 'Fill Blank Form' at exactly
the same second, so name collisions are rare, but not impossible.

In the exceptional case that directory names do collide, ODK Briefcase
then starts adding _1, _2, .etc to the end of the directory name, looking
for the first not-yet-used name.

This last step is where your 10+ minute load occurs -- after cycling
through and not deleting already-pulled forms from your phone, ODK
Briefcase gets clogged with duplicates of all that data, and the
test-and-increment loop takes forever to complete.

Mitch

On Fri, Mar 8, 2013 at 9:33 AM, Sam samantha.elghanayan@gmail.comwrote:

In pilot tests the past few weeks, my team has noticed that
the Briefcase "Pull" function takes longer with each new PDA we pull data
from. The first time we pull data from the ODK folder on a PDA, it takes
just a few seconds for the data to be downloaded. By the 10th PDA it takes
over 10 minutes.

This could certainly be bc we have a very long survey, and we accept
that we will face such hurdles. But if there's a way to minimize the time
it takes to pull data with Briefcase, please let us know.

Thanks,
Sam

--

Post: opendatakit@googlegroups.com
Unsubscribe: opendatakit+unsubscribe@googlegroups.com
Options: http://groups.google.com/group/opendatakit?hl=en


You received this message because you are subscribed to the Google
Groups "ODK Community" group.
To unsubscribe from this group and stop receiving emails from it, send
an email to opendatakit+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

--
Mitch Sundt
Software Engineer
University of Washington
mitchellsundt@gmail.com

--

Post: opendatakit@googlegroups.com
Unsubscribe: opendatakit+unsubscribe@googlegroups.com
Options: http://groups.google.com/group/opendatakit?hl=en


You received this message because you are subscribed to a topic in the
Google Groups "ODK Community" group.
To unsubscribe from this topic, visit
https://groups.google.com/d/topic/opendatakit/d-h4CY5z8Aw/unsubscribe?hl=en
.
To unsubscribe from this group and all its topics, send an email to
opendatakit+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

--
--
Post: opendatakit@googlegroups.com
Unsubscribe: opendatakit+unsubscribe@googlegroups.com
Options: http://groups.google.com/group/opendatakit?hl=en


You received this message because you are subscribed to the Google Groups
"ODK Community" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to opendatakit+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

--
Mitch Sundt
Software Engineer
University of Washington
mitchellsundt@gmail.com