[ODK Developers] Misplaced Responses

Hi all
I have a problem with my form: responses are being shifted to wrong places. Thus responses for variable A could be viewed under variable B. All this is happening when exporting the data using odk briefcase. Correct responses can be viewed in the submissions xml forms. What could be the cause of the problem? I am convinced that Iam using atleast higher versions of both odk collect and briefcase.

Thanks,
Humphreys

··· Mitch S wrote:

The correct solution is to define an in your form. That
instanceID will be used to merge the multiple submissions necessary to
upload the data. The form submission is already effectively a zip of the
data. The problem comes when that data is very large. The larger it is,
the more likely the transmission will fail, and the more resources are
needed on the server to handle the data in one pass. By explicitly
providing an instanceID, the data can be sent in smaller chunks and merged
on the server.

This mechanism is defined here:
https://bitbucket.org/javarosa/javarosa/wiki/OpenRosaMetaDataSchema
For ODK Aggregate, you don't need to specify the namespace, just have a
group in your form.

With ODK Collect 1.1.7 and later, the bind for the instanceID would be:

...






...




...

You can construct your own instanceID expressions. However, you should
avoid symbols and punctuation other than colons and dashes since the
parsing logic within Aggregate is likely fragile if you go wild with
punctuation (and that is used later on when retrieving images, repeat
groups, etc.). If you do construct your own values, they must be less than
80 characters long (a limitation of ODK Aggregate) and must be unique
across all devices and submissions.

Mitch

On Mon, Jun 4, 2012 at 3:42 PM, Kihara Absolomon soloincc@gmail.com wrote:

Hi Mitch,

Thanks for the heads up. Surprisingly I had never noticed this tab or just
didn't know its function prior to this! But we have some problems!

It seems that something weird happened during submissions. Under the
Submissions Admin Tab, I have 2 instances each of my un-submitted form. I
can see clearly that duplicate instances data is identical, down to the
auto inserted device-id, start-time and end-time. The only discrepancy
comes in the media data collected. Each instance is meant to have a minimum
of 5 photos and a no maximum, as there is a looping logic to add as many
photos as possible. These photos have been split between the duplicate
instances, in that some photos appear in one instance and not in the other
and vice versa. This is a perfect case of merging the duplicate instances.
I guess I have to do this at the DB level.

I don't know how many have experienced this scenario, but I think it
makes the case for the merge feature in case of in-complete submissions.

On another note, I tried using the "Manually Upload Submission Data"
function as a 2nd option, but it only offered 1 option of uploading the
media files. On using this feature, I ended up with an incomplete instance
again!

This can be solved by zipping all the media files and uploading them as
a zip file and doing the un-zipping and merging on the server side...just a
thought!

*
*
Regards
Kihara Absolomon

On Mon, Jun 4, 2012 at 9:05 PM, Mitch S mitchellsundt@gmail.com wrote:

On Aggregate, these submissions may have been determined to be incomplete
-- this occurs if the captured media attachments for the filled-in form
were not all successfully uploaded. You can look at incomplete submissions
on the Forms Management / Submission Admin tab. If you "Mark As Complete"
the form, that action will remove all traces of the missing attachments.
Alternatively, on that tab you can "Manually Upload Submission Data" and
navigate to the phone's /sdcard/odk/instances directory, into the directory
of the incomplete submission, and manually upload the submission's xml file
and missing media attachments to complete the submission.

Once the submission is complete, it will move to the Submissions tab and
disappear from the Submission Admin tab.

If you are using an older version of ODK Collect, it may be that the
media file formats were not recognized by that older version (1.1.5 and
earlier).

Mitch

On Mon, Jun 4, 2012 at 8:42 AM, Kihara Absolomon soloincc@gmail.comwrote:

Hi all,

I have some instances which were marked as finalized and ready to be
uploaded to the server. During the upload process there was some sort of
error and 6 of them weren't uploaded. I re-uploaded them and ODK reported
that they were uploaded successfully. However on logging in to aggregate,
the 6 instances are not reflecting in Aggregate and on Collect they are not
appearing as instances that can be uploaded. I can still see them under the
instances folder in the SD card.

How do I re-upload them to the server?

Regards
Kihara Absolomon

--
Mitch Sundt
Software Engineer
University of Washington
mitchellsundt@gmail.com

--
Mitch Sundt
Software Engineer
University of Washington
mitchellsundt@gmail.com

My wild guess is that ODK Briefcase has an older version of the form,
so it's definition of how to treat the data is wrong.

Please file a issue with the Form attached and submission instances
that demonstrate it is not being parsed correctly. By providing the
form and the submission instances that don't parse we will be able to
debug and fix the problem.

To file an issue please go to the following url and click on "New Issue".
http://code.google.com/p/opendatakit/issues/list
You can attach files to the issue description. Please attach the form
and example problem submission instances to the issue so we can
re-create the issue.

Cheers,
Waylon

··· On Mon, Jun 4, 2012 at 6:38 PM, Humphreys wrote: > Hi all > I have a problem with my form: responses are being shifted to wrong places. Thus responses for variable A could be viewed under variable B. All this is happening when exporting the data using odk briefcase. Correct responses can be viewed in the submissions xml forms. What could be the cause of the problem? I am convinced that Iam using atleast higher versions of both odk collect and briefcase. > > Thanks, > Humphreys > > Mitch S wrote: > >>The correct solution is to define an in your form. That >>instanceID will be used to merge the multiple submissions necessary to >>upload the data. The form submission is already effectively a zip of the >>data. The problem comes when that data is very large. The larger it is, >>the more likely the transmission will fail, and the more resources are >>needed on the server to handle the data in one pass. By explicitly >>providing an instanceID, the data can be sent in smaller chunks and merged >>on the server. >> >>This mechanism is defined here: >>https://bitbucket.org/javarosa/javarosa/wiki/OpenRosaMetaDataSchema >>For ODK Aggregate, you don't need to specify the namespace, just have a >> group in your form. >> >>With ODK Collect 1.1.7 and later, the bind for the instanceID would be: >> >>... >> >> >> >> >> >> >> ... >> >> >> >>>calculate="concat('uuid:', uuid())"/> >>... >> >>You can construct your own instanceID expressions. However, you should >>avoid symbols and punctuation other than colons and dashes since the >>parsing logic within Aggregate is likely fragile if you go wild with >>punctuation (and that is used later on when retrieving images, repeat >>groups, etc.). If you do construct your own values, they must be less than >>80 characters long (a limitation of ODK Aggregate) and must be unique >>across all devices and submissions. >> >>Mitch >> >>On Mon, Jun 4, 2012 at 3:42 PM, Kihara Absolomon wrote: >> >>> Hi Mitch, >>> >>> Thanks for the heads up. Surprisingly I had never noticed this tab or just >>> didn't know its function prior to this! But we have some problems! >>> >>> It seems that something weird happened during submissions. Under the >>> Submissions Admin Tab, I have 2 instances each of my un-submitted form. I >>> can see clearly that duplicate instances data is identical, down to the >>> auto inserted device-id, start-time and end-time. The only discrepancy >>> comes in the media data collected. Each instance is meant to have a minimum >>> of 5 photos and a no maximum, as there is a looping logic to add as many >>> photos as possible. These photos have been split between the duplicate >>> instances, in that some photos appear in one instance and not in the other >>> and vice versa. This is a perfect case of merging the duplicate instances. >>> I guess I have to do this at the DB level. >>> >>> *I don't know how many have experienced this scenario, but I think it >>> makes the case for the merge feature in case of in-complete submissions.* >>> >>> On another note, I tried using the "Manually Upload Submission Data" >>> function as a 2nd option, but it only offered 1 option of uploading the >>> media files. On using this feature, I ended up with an incomplete instance >>> again! >>> >>> *This can be solved by zipping all the media files and uploading them as >>> a zip file and doing the un-zipping and merging on the server side...just a >>> thought!* >>> * >>> * >>> Regards >>> Kihara Absolomon >>> >>> >>> >>> On Mon, Jun 4, 2012 at 9:05 PM, Mitch S wrote: >>> >>>> On Aggregate, these submissions may have been determined to be incomplete >>>> -- this occurs if the captured media attachments for the filled-in form >>>> were not all successfully uploaded. You can look at incomplete submissions >>>> on the Forms Management / Submission Admin tab. If you "Mark As Complete" >>>> the form, that action will remove all traces of the missing attachments. >>>> Alternatively, on that tab you can "Manually Upload Submission Data" and >>>> navigate to the phone's /sdcard/odk/instances directory, into the directory >>>> of the incomplete submission, and manually upload the submission's xml file >>>> and missing media attachments to complete the submission. >>>> >>>> Once the submission is complete, it will move to the Submissions tab and >>>> disappear from the Submission Admin tab. >>>> >>>> If you are using an older version of ODK Collect, it may be that the >>>> media file formats were not recognized by that older version (1.1.5 and >>>> earlier). >>>> >>>> Mitch >>>> >>>> >>>> On Mon, Jun 4, 2012 at 8:42 AM, Kihara Absolomon wrote: >>>> >>>>> Hi all, >>>>> >>>>> I have some instances which were marked as finalized and ready to be >>>>> uploaded to the server. During the upload process there was some sort of >>>>> error and 6 of them weren't uploaded. I re-uploaded them and ODK reported >>>>> that they were uploaded successfully. However on logging in to aggregate, >>>>> the 6 instances are not reflecting in Aggregate and on Collect they are not >>>>> appearing as instances that can be uploaded. I can still see them under the >>>>> instances folder in the SD card. >>>>> >>>>> How do I re-upload them to the server? >>>>> >>>>> Regards >>>>> Kihara Absolomon >>>>> >>>> >>>> >>>> >>>> -- >>>> Mitch Sundt >>>> Software Engineer >>>> University of Washington >>>> mitchellsundt@gmail.com >>>> >>> >>> >> >> >>-- >>Mitch Sundt >>Software Engineer >>University of Washington >>mitchellsundt@gmail.com

Humphreys,

How common is the error? I've seen this variable shifting happen about once
every 200 surveys. We have attributed this error to an issue with the
MyTouch 3G phones, which have a little memory glitch.

☞§※☼:airplane::open_umbrella::slight_smile:
~Neil

··· On Mon, Jun 4, 2012 at 9:38 PM, Humphreys wrote:

Hi all
I have a problem with my form: responses are being shifted to wrong
places. Thus responses for variable A could be viewed under variable B.
All this is happening when exporting the data using odk briefcase. Correct
responses can be viewed in the submissions xml forms. What could be the
cause of the problem? I am convinced that Iam using atleast higher versions
of both odk collect and briefcase.

Thanks,
Humphreys

There are two issues here --

(1) the reported issue of the data being properly recorded in ODK Aggregate
/ ODK Collect, but not being exported properly in ODK Briefcase
(2) possible errors on the phone, which are very rare.

This is the first I've heard of (1).

Mitch

··· On Tue, Jun 5, 2012 at 8:58 AM, Neil Hendrick wrote:

Humphreys,

How common is the error? I've seen this variable shifting happen about
once every 200 surveys. We have attributed this error to an issue with the
MyTouch 3G phones, which have a little memory glitch.

☞§※☼:airplane::open_umbrella::slight_smile:
~Neil

On Mon, Jun 4, 2012 at 9:38 PM, Humphreys humphkabota@gmail.com wrote:

Hi all
I have a problem with my form: responses are being shifted to wrong
places. Thus responses for variable A could be viewed under variable B.
All this is happening when exporting the data using odk briefcase. Correct
responses can be viewed in the submissions xml forms. What could be the
cause of the problem? I am convinced that Iam using atleast higher versions
of both odk collect and briefcase.

Thanks,
Humphreys

--
Mitch Sundt
Software Engineer
University of Washington
mitchellsundt@gmail.com