I've been using ODK for over a year now on several projects with great
success. We have a project we just rolled out and we are having problems
for the first time. There are no unique items in this current deployment
other than each form has 3 required photos. The devices being used are
ASUS Transformers with 8MP camera. I fear the image sizes may be to blame,
and I've dialed down the image quality on the devices. However, I still
need to pull the data that has been collected to date.
The issue is when the field teams submit, they get the success message, yet
not all of their submissions are seen in ODK Aggregate web interface. At
this point, 12 of 192 records fail to display in Aggregate, or Pull/Export
from Briefcase. Upon further investigation, all of the records missing are
actually in postgres but they are duplicated records?
I found in another post about ODK Collect limitations when it comes to
submissions and forms that exceed 10MB. That the submissions often get
broken into multiple records. This seems to be our issue. How can I get
the csv and associate photos to pull/export from Briefcase. The only other
option is to manually extract the submitted forms from the tablets, but
this is not ideal.
Your forms likely do not have an instanceID metadata field. Submissions to
ODK Aggregate are split at 10MB to ensure that they get transmitted and
processed on ODK Aggregate within the 60-second limit imposed by Google
AppEngine. With larger images, and with 8 images per submission, you are
undoubtedly having your submissions fragmented into these 10MB chunks
during the submission process. In order to reconstruct the single
submission from these fragments, the forms MUST contain an instanceID
metadata field.
If they don't, then each fragment is saved as a separate instance on ODK
Aggregate, and will be flagged as incomplete because the entire set of
media files (images) are not attached to each of the fragments -- each
fragment consists of the submission and the few image files that fit within
a 10MB transmission limit.
The incomplete submissions will be present on the Form Management /
Submission Admin sub-tab. This is where incompletely-transmitted data
resides.
Unfortunately, there is no easy way to resolve this. If you do your data
analysis in a downstream system, you can mark-as-complete all these
incomplete submissions (which will eliminate the missing attachments from
the submission and mark the submission as complete). Then, in the
downstream system, you would need to consolidate the different fragmented
submissions into a single composite record.
Mitch
···
On Fri, Jan 25, 2013 at 8:41 AM, Jason Wheatley wrote:
Hello all,
I've been using ODK for over a year now on several projects with great
success. We have a project we just rolled out and we are having problems
for the first time. There are no unique items in this current deployment
other than each form has 3 required photos. The devices being used are
ASUS Transformers with 8MP camera. I fear the image sizes may be to blame,
and I've dialed down the image quality on the devices. However, I still
need to pull the data that has been collected to date.
The issue is when the field teams submit, they get the success message,
yet not all of their submissions are seen in ODK Aggregate web interface.
At this point, 12 of 192 records fail to display in Aggregate, or
Pull/Export from Briefcase. Upon further investigation, all of the records
missing are actually in postgres but they are duplicated records?
I found in another post about ODK Collect limitations when it comes to
submissions and forms that exceed 10MB. That the submissions often get
broken into multiple records. This seems to be our issue. How can I get
the csv and associate photos to pull/export from Briefcase. The only other
option is to manually extract the submitted forms from the tablets, but
this is not ideal.
Thank you so much for the reply. I see this is an issue that has been
repeated several time on the board here. I apologize for not locating it
before posting it. I recreated the form, and used ODK XLSForm utility to
convert to xml. Before I was using formhub, and it appears formhub does
not add the instanceid meta at this time.
I tested several submissions with photos totaling 20MB and Briefcase had no
problems pulling or exporting the records.
Now I just have manually process the previous submissions from the devices
in order to reclaim the previously "dropped" submissions.
I'm just glad we caught it early on in the project.
Thanks again.
···
On Fri, Jan 25, 2013 at 5:07 PM, Mitch S wrote:
Your forms likely do not have an instanceID metadata field. Submissions to
ODK Aggregate are split at 10MB to ensure that they get transmitted and
processed on ODK Aggregate within the 60-second limit imposed by Google
AppEngine. With larger images, and with 8 images per submission, you are
undoubtedly having your submissions fragmented into these 10MB chunks
during the submission process. In order to reconstruct the single
submission from these fragments, the forms MUST contain an instanceID
metadata field.
If they don't, then each fragment is saved as a separate instance on ODK
Aggregate, and will be flagged as incomplete because the entire set of
media files (images) are not attached to each of the fragments -- each
fragment consists of the submission and the few image files that fit within
a 10MB transmission limit.
The incomplete submissions will be present on the Form Management /
Submission Admin sub-tab. This is where incompletely-transmitted data
resides.
Unfortunately, there is no easy way to resolve this. If you do your data
analysis in a downstream system, you can mark-as-complete all these
incomplete submissions (which will eliminate the missing attachments from
the submission and mark the submission as complete). Then, in the
downstream system, you would need to consolidate the different fragmented
submissions into a single composite record.
I've been using ODK for over a year now on several projects with great
success. We have a project we just rolled out and we are having problems
for the first time. There are no unique items in this current deployment
other than each form has 3 required photos. The devices being used are
ASUS Transformers with 8MP camera. I fear the image sizes may be to blame,
and I've dialed down the image quality on the devices. However, I still
need to pull the data that has been collected to date.
The issue is when the field teams submit, they get the success message,
yet not all of their submissions are seen in ODK Aggregate web interface.
At this point, 12 of 192 records fail to display in Aggregate, or
Pull/Export from Briefcase. Upon further investigation, all of the records
missing are actually in postgres but they are duplicated records?
I found in another post about ODK Collect limitations when it comes to
submissions and forms that exceed 10MB. That the submissions often get
broken into multiple records. This seems to be our issue. How can I get
the csv and associate photos to pull/export from Briefcase. The only other
option is to manually extract the submitted forms from the tablets, but
this is not ideal.