Descrepancy between ODK Aggregate & BriefCase total records

Support
1

Hi, I'm curious if anyone has experienced this or can explain it. In my
Excel spreadsheet (from ODK Aggregate csv export), one data collector has
164 web submissions via the KoboCollect to ODK Aggregate. But when I got
his phone back at the end of the survey, I did a Briefcase import (Pull)
and it imported 170 submissions - a difference of 6 entries.

So I did a duplicate search on the Latitude field and start date -- none
found!!. Next, I grabbed the Latitude field from the web submissions
(KoboCollect to ODK Aggregate) and pasted it into the Latitude field in the
BriefCase csv (Latitude columun) and then sorted them by Latitude. And I
saw the 6 BriefCase import entries that had no matching web submissions.
Any ideas? The data looks good and definitely these 6 submissions SHOULD be
in the system.

~DataMax

NOTE1: One FAILED submission that was left on the phone was successfully
imported by BriefCase.

NOTE2: The data collectors are instructed NOT to DELETE anything (saved
forms, submitted forms, etc.) -- just in case!

NOTE3: BriefCase error/warning message at startup. It still imported 170
forms without any problems though. Could this be why I was having duplicate
submissions that I raised in another thread?

··· -------------------------- C:\Program Files (x86)\Java\jre6\bin>java -jar briefcase.jar 16-Jun-2012 12:14:42 PM org.opendatakit.briefcase.ui.FormTransferTable$DetailBut ton INFO: creating details button 16-Jun-2012 12:14:42 PM org.opendatakit.briefcase.ui.FormTransferTable$DetailBut ton INFO: creating details button Parsing form... Title: "Marys Survey" Warning: 1 Unrecognized attributes found in Element [bind] and will be ignored: [format] Location: 
Problem found at nodeset: /html/head/model/bind
With element <bind id="GPS" nodeset="/Marys_Survey/GPS" format=

"gps" type="geopoint" required="true()"/>

Warning: 1 Unrecognized attributes found in Element [bind] and will be
ignored:
[format] Location:

Problem found at nodeset: /html/head/model/bind
With element <bind id="Pic" nodeset="/Marys_Survey/Pic" format=

"image" type="binary" required="true()"/>

XForm Parse WARNING: unrecognized data type
XForm Parse WARNING: unrecognized data type
Warning: 1 Unrecognized attributes found in Element [upload] and will be
ignored
: [mediatype] Location:

Problem found at nodeset: /html/body/upload
With element <upload bind="Pic" mediatype="image/*">

Warning: 1 Unrecognized attributes found in Element [upload] and will be
ignored
: [bind] Location:

Problem found at nodeset: /html/body/upload
With element <upload bind="Pic" mediatype="image/*">

Warning: 1 Unrecognized attributes found in Element [group] and will be
ignored:
[id] Location:

Problem found at nodeset: /html/body/group
With element <group id="INTERESTS">

Warning: 1 Unrecognized attributes found in Element [group] and will be
ignored:
[id] Location:

Problem found at nodeset: /html/body/group
With element <group id="INTERESTS2">

Warning: 1 Unrecognized attributes found in Element [group] and will be
ignored:
[id] Location:

Problem found at nodeset: /html/body/group
With element <group id="INTERESTS3">

Warning: 1 Unrecognized attributes found in Element [Marys_Survey]
and will be ignored: [id] Location:

Problem found at nodeset: /html/head/model/instance/Marys_Survey
With element <Marys_Survey id="Marys_Survey">

16-Jun-2012 12:15:36 PM
org.opendatakit.briefcase.ui.FormTransferTable$DetailBut
ton
INFO: creating details button

2

Max,

While I am not familiar with all the differences that exist between
KoboCollect and ODK Collect as there could be an issue with
KoboCollect that I am not aware of, so this answer is more based on
ODK Collect (but they are normally very similar).

My first guess to your question is the fact that the form submission
were not finalized and therefore Collect never submitted them to
Aggregate. Collect maintains state in it's database on what is still a
work in progress, what has been finalized and not submitted, and what
has already been submitted. Briefcase on the other hand just looks at
the storage files on the Android devices file system and grabs them
all (does not have access to collect state information). To make sure
people don't lose work Collect rarely deletes data files enabling an
administrator to use something like Briefcase to recover all the work
that went on with the phone. Therefore the models/methods of Collect
posting to a server using state information provided by the user
versus Briefcase grabbing every file it can ignoring collect state
information are different and therefore will have different results.
The question why didn't the submissions you are referring to not get
submitted is a vaild one but it highly depends on what state the
submissions were set to by Collect (by the user). Again my first guess
is the submssions were not set to finalized so Collect assumed the
user wanted to make more revisions before sending the submission to
Aggregate (or another server). The only way to know for sure is to
know what state Collect had those submissions in.

Waylon

··· On Sun, Jun 17, 2012 at 8:37 PM, DataMax wrote: > Hi, I'm curious if anyone has experienced this or can explain it. In my > Excel spreadsheet (from ODK Aggregate csv export), one data collector has > 164 web submissions via the KoboCollect to ODK Aggregate. But when I got his > phone back at the end of the survey, I did a Briefcase import (Pull) and it > imported 170 submissions - a difference of 6 entries. > > So I did a duplicate search on the Latitude field and start date -- none > found!!. Next, I grabbed the Latitude field from the web submissions > (KoboCollect to ODK Aggregate) and pasted it into the Latitude field in the > BriefCase csv (Latitude columun) and then sorted them by Latitude. And I saw > the 6 BriefCase import entries that had no matching web submissions. Any > ideas? The data looks good and definitely these 6 submissions SHOULD be in > the system. > > ~DataMax > > NOTE1: One FAILED submission that was left on the phone was successfully > imported by BriefCase. > > NOTE2: The data collectors are instructed NOT to DELETE anything (saved > forms, submitted forms, etc.) -- just in case! > > NOTE3: BriefCase error/warning message at startup. It still imported 170 > forms without any problems though. Could this be why I was having duplicate > submissions that I raised in another thread? > > -------------------------- > C:\Program Files (x86)\Java\jre6\bin>java -jar briefcase.jar > 16-Jun-2012 12:14:42 PM > org.opendatakit.briefcase.ui.FormTransferTable$DetailBut > ton > INFO: creating details button > 16-Jun-2012 12:14:42 PM > org.opendatakit.briefcase.ui.FormTransferTable$DetailBut > ton > INFO: creating details button > Parsing form... > Title: "Marys Survey" > Warning: 1 Unrecognized attributes found in Element [bind] and will be > ignored: > [format] Location: > > Problem found at nodeset: /html/head/model/bind > With element "gps" type="geopoint" required="true()"/> > > Warning: 1 Unrecognized attributes found in Element [bind] and will be > ignored: > [format] Location: > > Problem found at nodeset: /html/head/model/bind > With element "image" type="binary" required="true()"/> > > XForm Parse WARNING: unrecognized data type [] > XForm Parse WARNING: unrecognized data type [] > Warning: 1 Unrecognized attributes found in Element [upload] and will be > ignored > : [mediatype] Location: > > Problem found at nodeset: /html/body/upload > With element > > Warning: 1 Unrecognized attributes found in Element [upload] and will be > ignored > : [bind] Location: > > Problem found at nodeset: /html/body/upload > With element > > Warning: 1 Unrecognized attributes found in Element [group] and will be > ignored: > [id] Location: > > Problem found at nodeset: /html/body/group > With element > > Warning: 1 Unrecognized attributes found in Element [group] and will be > ignored: > [id] Location: > > Problem found at nodeset: /html/body/group > With element > > Warning: 1 Unrecognized attributes found in Element [group] and will be > ignored: > [id] Location: > > Problem found at nodeset: /html/body/group > With element > > Warning: 1 Unrecognized attributes found in Element [Marys_Survey] > and will be ignored: [id] Location: > > Problem found at nodeset: /html/head/model/instance/Marys_Survey > With element > > 16-Jun-2012 12:15:36 PM > org.opendatakit.briefcase.ui.FormTransferTable$DetailBut > ton > INFO: creating details button > -------------------------- > > > > > > > -- > Post: opendatakit@googlegroups.com > Unsubscribe: opendatakit+unsubscribe@googlegroups.com > Options: http://groups.google.com/group/opendatakit?hl=en
3

Additionally, please look at the Form Management / Submission Admin tab.

It may be that some uploads to the server failed to complete. If that is
the case, the partially-uploaded submissions will appear on the above tab.
This happens if the submission had media attachments and one or more of
them could not be uploaded to the server (e.g., due to a loss of wireless
connectivity during the transmission).

Once the submission and missing attachments are uploaded, (you can do this
with the "Manually Upload Submission Data" button on this tab), the
submission will be moved to the Submissions tab and will appear in the CSV
export, Briefcase, etc.

Mitch

··· On Mon, Jun 18, 2012 at 10:53 AM, W. Brunette wrote:

Max,

While I am not familiar with all the differences that exist between
KoboCollect and ODK Collect as there could be an issue with
KoboCollect that I am not aware of, so this answer is more based on
ODK Collect (but they are normally very similar).

My first guess to your question is the fact that the form submission
were not finalized and therefore Collect never submitted them to
Aggregate. Collect maintains state in it's database on what is still a
work in progress, what has been finalized and not submitted, and what
has already been submitted. Briefcase on the other hand just looks at
the storage files on the Android devices file system and grabs them
all (does not have access to collect state information). To make sure
people don't lose work Collect rarely deletes data files enabling an
administrator to use something like Briefcase to recover all the work
that went on with the phone. Therefore the models/methods of Collect
posting to a server using state information provided by the user
versus Briefcase grabbing every file it can ignoring collect state
information are different and therefore will have different results.
The question why didn't the submissions you are referring to not get
submitted is a vaild one but it highly depends on what state the
submissions were set to by Collect (by the user). Again my first guess
is the submssions were not set to finalized so Collect assumed the
user wanted to make more revisions before sending the submission to
Aggregate (or another server). The only way to know for sure is to
know what state Collect had those submissions in.

Waylon

On Sun, Jun 17, 2012 at 8:37 PM, DataMax maxtheitpro@gmail.com wrote:

Hi, I'm curious if anyone has experienced this or can explain it. In my
Excel spreadsheet (from ODK Aggregate csv export), one data collector has
164 web submissions via the KoboCollect to ODK Aggregate. But when I got
his
phone back at the end of the survey, I did a Briefcase import (Pull) and
it
imported 170 submissions - a difference of 6 entries.

So I did a duplicate search on the Latitude field and start date -- none
found!!. Next, I grabbed the Latitude field from the web submissions
(KoboCollect to ODK Aggregate) and pasted it into the Latitude field in
the
BriefCase csv (Latitude columun) and then sorted them by Latitude. And I
saw
the 6 BriefCase import entries that had no matching web submissions. Any
ideas? The data looks good and definitely these 6 submissions SHOULD be
in
the system.

~DataMax

NOTE1: One FAILED submission that was left on the phone was successfully
imported by BriefCase.

NOTE2: The data collectors are instructed NOT to DELETE anything (saved
forms, submitted forms, etc.) -- just in case!

NOTE3: BriefCase error/warning message at startup. It still imported 170
forms without any problems though. Could this be why I was having
duplicate
submissions that I raised in another thread?

C:\Program Files (x86)\Java\jre6\bin>java -jar briefcase.jar
16-Jun-2012 12:14:42 PM
org.opendatakit.briefcase.ui.FormTransferTable$DetailBut
ton
INFO: creating details button
16-Jun-2012 12:14:42 PM
org.opendatakit.briefcase.ui.FormTransferTable$DetailBut
ton
INFO: creating details button
Parsing form...
Title: "Marys Survey"
Warning: 1 Unrecognized attributes found in Element [bind] and will be
ignored:
[format] Location:

Problem found at nodeset: /html/head/model/bind
With element <bind id="GPS" nodeset="/Marys_Survey/GPS" format=

"gps" type="geopoint" required="true()"/>

Warning: 1 Unrecognized attributes found in Element [bind] and will be
ignored:
[format] Location:

Problem found at nodeset: /html/head/model/bind
With element <bind id="Pic" nodeset="/Marys_Survey/Pic" format=

"image" type="binary" required="true()"/>

XForm Parse WARNING: unrecognized data type
XForm Parse WARNING: unrecognized data type
Warning: 1 Unrecognized attributes found in Element [upload] and will be
ignored
: [mediatype] Location:

Problem found at nodeset: /html/body/upload
With element <upload bind="Pic" mediatype="image/*">

Warning: 1 Unrecognized attributes found in Element [upload] and will be
ignored
: [bind] Location:

Problem found at nodeset: /html/body/upload
With element <upload bind="Pic" mediatype="image/*">

Warning: 1 Unrecognized attributes found in Element [group] and will be
ignored:
[id] Location:

Problem found at nodeset: /html/body/group
With element <group id="INTERESTS">

Warning: 1 Unrecognized attributes found in Element [group] and will be
ignored:
[id] Location:

Problem found at nodeset: /html/body/group
With element <group id="INTERESTS2">

Warning: 1 Unrecognized attributes found in Element [group] and will be
ignored:
[id] Location:

Problem found at nodeset: /html/body/group
With element <group id="INTERESTS3">

Warning: 1 Unrecognized attributes found in Element [Marys_Survey]
and will be ignored: [id] Location:

Problem found at nodeset: /html/head/model/instance/Marys_Survey
With element <Marys_Survey id="Marys_Survey">

16-Jun-2012 12:15:36 PM
org.opendatakit.briefcase.ui.FormTransferTable$DetailBut
ton
INFO: creating details button

--
Post: opendatakit@googlegroups.com
Unsubscribe: opendatakit+unsubscribe@googlegroups.com
Options: http://groups.google.com/group/opendatakit?hl=en

--
Post: opendatakit@googlegroups.com
Unsubscribe: opendatakit+unsubscribe@googlegroups.com
Options: http://groups.google.com/group/opendatakit?hl=en

--
Mitch Sundt
Software Engineer
University of Washington
mitchellsundt@gmail.com