Finalizing encrypted surveys

Support
1

Hi all,

We are currently having issues with the briefcase in downloading all the
surveys. Between the aggregate and briefcase, there are about 150 surveys
that are not being exported. Since the surveys have signatures that are
stored as media files - and if data collectors did not get consent, there
are no media files for those submitted surveys. I read through other
questions by people, and it seems that we have to somehow complete the form
in aggregate in order for the briefcase to export these files. Since the
forms are encrypted, I am not sure how to mark them as complete.

Any thoughts on how I can do this? Or what's the best way for us to
continue to download all data using the briefcase?

Thanks
Shreya

2

Yikes. With encrypted forms, do not click the "Mark as Complete" button!!!

This indicates a problem with the transmission of data off the device into
ODK Aggregate.

If there should be an image, but the image was never received by ODK
Aggregate, then the submission will appear in the Form Management /
Submission Admin sub-tab until the missing image is sent to the server.
You should generally NOT click the "Mark As Complete" button. Instead, you
should examine the device and determine why it did not submit the image to
the server.

The device should have reported a transmission failure and left the
submission as "Failed to send". When you re-send it, it would re-try to
send the image, and, upon receipt, the completed submission would move from
the Submission Admin tab to the Submissions tab.

As always, ensure that you have the very latest ODK Collect. If you are
using a custom build, there was an error in an earlier custom Google Earth
Outreach build that could have caused the erroneous deletion of submissions
off the device when the "automatically delete submissions after
transmission" option was enabled. To be safe, uncheck that until you can
confirm that your version of ODK Collect does not have that flaw (the flaw
did not make it into the main code tree).

When working with encrypted forms, the clicking of the "Mark As Complete"
button to force the submission into the Submissions tab would likely cause
the failure you reported in Briefcase when you try to Export to CSV (and
data corruption of all subsequent media attachments). I never considered
the interactions between encryption and the "Mark as Complete" action.

So, in conclusion, examine the devices and diagnose why they are not
submitting their images of the signatures.

For encrypted forms, avoid using the "Mark as Complete" button, as it will
likely corrupt all media attachments.

Mitch

··· On Fri, Oct 31, 2014 at 6:56 AM, Shreya Agrawal wrote:

Hi all,

We are currently having issues with the briefcase in downloading all the
surveys. Between the aggregate and briefcase, there are about 150 surveys
that are not being exported. Since the surveys have signatures that are
stored as media files - and if data collectors did not get consent, there
are no media files for those submitted surveys. I read through other
questions by people, and it seems that we have to somehow complete the form
in aggregate in order for the briefcase to export these files. Since the
forms are encrypted, I am not sure how to mark them as complete.

Any thoughts on how I can do this? Or what's the best way for us to
continue to download all data using the briefcase?

Thanks
Shreya

--

Post: opendatakit@googlegroups.com
Unsubscribe: opendatakit+unsubscribe@googlegroups.com
Options: http://groups.google.com/group/opendatakit?hl=en

You received this message because you are subscribed to the Google Groups
"ODK Community" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to opendatakit+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--
Mitch Sundt
Software Engineer
University of Washington
mitchellsundt@gmail.com

3

Hey Mitch,

Quick clarification: the signature files don't exist for these surveys because we never captured them. If the person does not give consent, we don't have a signature so there are no image files. Therefore, when we are downloading the data, all these files don't have an image. Is that the reason for the error?

If we don't Mark as complete - how do the data collectors upload the data? These are the files that are finalized?

I wanted to know how we can still download all these surveys that don't have an image through the briefcase without the error. Is that possible?

Also, we have the latest version of ODK Downloaded that's availble on the Google store.

Can you please clarify?

Thanks
Shreya

4

No. Not having captured a signature would not be the reason for the error.
But if you did capture an image, but it was not preserved through the
submission process, then that would be a problem.

The "Mark as Complete" button is on the ODK Aggregate server, under the
Forms Management / Submissions Admin tab; it IS NOT the "Finalize" button
on ODK Collect.

My comment refers specifically to any submissions displayed under the Forms
Management / Submissions Admin tab. If you have submissions displaying on
that tab, then they are missing media attachments (e.g., signature files
that were captured, but failed to be sent to the server) -- the sending of
the submission to the server somehow failed. The user on the device will
need to re-send those submissions and their attachments. Choosing "Mark as
Complete" will cause problems during the decryption of the affected
submission.

If your briefcase storage location does have submissions that are missing
media attachments (and it sounds like it does), I believe it will affect
just those submissions, and not the pulling of data or the generation of
the full csv file of all results -- but I have not tested this. Please
confirm that this is true through your own testing. I do not know if the
affected data will be emitted into the csv or not. I think it is likely
that the affected submissions will not have their data written to the csv,
but I have not confirmed this.

··· On Sat, Nov 1, 2014 at 12:03 PM, Shreya Agrawal wrote:

Hey Mitch,

Quick clarification: the signature files don't exist for these surveys
because we never captured them. If the person does not give consent, we
don't have a signature so there are no image files. Therefore, when we are
downloading the data, all these files don't have an image. Is that the
reason for the error?

If we don't Mark as complete - how do the data collectors upload the data?
These are the files that are finalized?

I wanted to know how we can still download all these surveys that don't
have an image through the briefcase without the error. Is that possible?

Also, we have the latest version of ODK Downloaded that's availble on the
Google store.

Can you please clarify?

Thanks
Shreya

--

Post: opendatakit@googlegroups.com
Unsubscribe: opendatakit+unsubscribe@googlegroups.com
Options: http://groups.google.com/group/opendatakit?hl=en

You received this message because you are subscribed to the Google Groups
"ODK Community" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to opendatakit+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--
Mitch Sundt
Software Engineer
University of Washington
mitchellsundt@gmail.com

5

Hey Mitch,

Thanks a lot. This is much more clear now.

I went back and double checked a few things now:

  1. ODK Aggregate/Form Management/Submission Admin tab only has 2 surveys
    that are there. I will not mark them as complete and leave them as is.
  2. Briefcase is downloading data for those files that don't have an image.
    So we have 1,743 submissions and only 1,600 images. So that's not an
    issue
  3. Briefcase does continue to download other data (the new one), but as
    soon as we hit the error for the particular image, it stops (its the same
    unique number ID that keeps coming up). So we have about 151 surveys after
    that particular image that we cannot download. I can see all the files
    that have been submitted on ODK Aggregate, but they are all encrypted. So I
    have no idea which image ODK Briefcase is referring to and how I can
    manually delete it from the Briefcase so that I can still access the 151
    surveys.

Any thoughts?

Thanks

··· On Monday, November 3, 2014 2:33:18 PM UTC-5, Mitch Sundt wrote: > > No. Not having captured a signature would not be the reason for the > error. But if you did capture an image, but it was not preserved through > the submission process, then that would be a problem. > > The "Mark as Complete" button is on the ODK Aggregate server, under the > Forms Management / Submissions Admin tab; it IS NOT the "Finalize" button > on ODK Collect. > > My comment refers specifically to any submissions displayed under the > Forms Management / Submissions Admin tab. If you have submissions > displaying on that tab, then they are missing media attachments (e.g., > signature files that were captured, but failed to be sent to the server) -- > the sending of the submission to the server somehow failed. The user on the > device will need to re-send those submissions and their attachments. > Choosing "Mark as Complete" will cause problems during the decryption of > the affected submission. > > If your briefcase storage location does have submissions that are missing > media attachments (and it sounds like it does), I believe it will affect > just those submissions, and not the pulling of data or the generation of > the full csv file of all results -- but I have not tested this. Please > confirm that this is true through your own testing. I do not know if the > affected data will be emitted into the csv or not. I think it is likely > that the affected submissions will not have their data written to the csv, > but I have not confirmed this. > > > > > > > > > > > On Sat, Nov 1, 2014 at 12:03 PM, Shreya Agrawal <s.agra...@gmail.com > wrote: > >> Hey Mitch, >> >> Quick clarification: the signature files don't exist for these surveys >> because we never captured them. If the person does not give consent, we >> don't have a signature so there are no image files. Therefore, when we are >> downloading the data, all these files don't have an image. Is that the >> reason for the error? >> >> If we don't Mark as complete - how do the data collectors upload the >> data? These are the files that are finalized? >> >> I wanted to know how we can still download all these surveys that don't >> have an image through the briefcase without the error. Is that possible? >> >> Also, we have the latest version of ODK Downloaded that's availble on the >> Google store. >> >> Can you please clarify? >> >> Thanks >> Shreya >> >> -- >> -- >> Post: opend...@googlegroups.com >> Unsubscribe: opendatakit...@googlegroups.com >> Options: http://groups.google.com/group/opendatakit?hl=en >> >> --- >> You received this message because you are subscribed to the Google Groups >> "ODK Community" group. >> To unsubscribe from this group and stop receiving emails from it, send an >> email to opendatakit...@googlegroups.com . >> For more options, visit https://groups.google.com/d/optout. >> > > > > -- > Mitch Sundt > Software Engineer > University of Washington > mitche...@gmail.com >
6

I will be able to work on this next week, and can send you a new ODK
Briefcase without this issue (it might omit the problematic survey from the
csv, but will not cause the csv output to stop).

··· On Tue, Nov 4, 2014 at 8:14 AM, Shreya Agrawal wrote:

Hey Mitch,

Thanks a lot. This is much more clear now.

I went back and double checked a few things now:

  1. ODK Aggregate/Form Management/Submission Admin tab only has 2 surveys
    that are there. I will not mark them as complete and leave them as is.
  2. Briefcase is downloading data for those files that don't have an
    image. So we have 1,743 submissions and only 1,600 images. So that's not
    an issue
  3. Briefcase does continue to download other data (the new one), but as
    soon as we hit the error for the particular image, it stops (its the same
    unique number ID that keeps coming up). So we have about 151 surveys after
    that particular image that we cannot download. I can see all the files
    that have been submitted on ODK Aggregate, but they are all encrypted. So I
    have no idea which image ODK Briefcase is referring to and how I can
    manually delete it from the Briefcase so that I can still access the 151
    surveys.

Any thoughts?

Thanks

On Monday, November 3, 2014 2:33:18 PM UTC-5, Mitch Sundt wrote:

No. Not having captured a signature would not be the reason for the
error. But if you did capture an image, but it was not preserved through
the submission process, then that would be a problem.

The "Mark as Complete" button is on the ODK Aggregate server, under the
Forms Management / Submissions Admin tab; it IS NOT the "Finalize" button
on ODK Collect.

My comment refers specifically to any submissions displayed under the
Forms Management / Submissions Admin tab. If you have submissions
displaying on that tab, then they are missing media attachments (e.g.,
signature files that were captured, but failed to be sent to the server) --
the sending of the submission to the server somehow failed. The user on the
device will need to re-send those submissions and their attachments.
Choosing "Mark as Complete" will cause problems during the decryption of
the affected submission.

If your briefcase storage location does have submissions that are missing
media attachments (and it sounds like it does), I believe it will affect
just those submissions, and not the pulling of data or the generation of
the full csv file of all results -- but I have not tested this. Please
confirm that this is true through your own testing. I do not know if the
affected data will be emitted into the csv or not. I think it is likely
that the affected submissions will not have their data written to the csv,
but I have not confirmed this.

On Sat, Nov 1, 2014 at 12:03 PM, Shreya Agrawal s.agra...@gmail.com wrote:

Hey Mitch,

Quick clarification: the signature files don't exist for these surveys
because we never captured them. If the person does not give consent, we
don't have a signature so there are no image files. Therefore, when we are
downloading the data, all these files don't have an image. Is that the
reason for the error?

If we don't Mark as complete - how do the data collectors upload the
data? These are the files that are finalized?

I wanted to know how we can still download all these surveys that don't
have an image through the briefcase without the error. Is that possible?

Also, we have the latest version of ODK Downloaded that's availble on
the Google store.

Can you please clarify?

Thanks
Shreya

--

Post: opend...@googlegroups.com
Unsubscribe: opendatakit...@googlegroups.com
Options: http://groups.google.com/group/opendatakit?hl=en

You received this message because you are subscribed to the Google
Groups "ODK Community" group.
To unsubscribe from this group and stop receiving emails from it, send
an email to opendatakit...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--
Mitch Sundt
Software Engineer
University of Washington
mitche...@gmail.com

--

Post: opendatakit@googlegroups.com
Unsubscribe: opendatakit+unsubscribe@googlegroups.com
Options: http://groups.google.com/group/opendatakit?hl=en

You received this message because you are subscribed to the Google Groups
"ODK Community" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to opendatakit+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--
Mitch Sundt
Software Engineer
University of Washington
mitchellsundt@gmail.com

7

Thanks Mitch! Should we connect via email on this?

Also, if you fix the Briefcase, will that fix this problem and we can have
access to rest of our data? If you think it will be a reoccurring problem,
we can create a new survey for data collectors to start using so that the
rest of the data does not get corrupted. I am just trying to understand
the severity of the problem.

Thanks!

··· On Tuesday, November 4, 2014 6:39:16 PM UTC-5, Mitch Sundt wrote: > > I will be able to work on this next week, and can send you a new ODK > Briefcase without this issue (it might omit the problematic survey from the > csv, but will not cause the csv output to stop). > > On Tue, Nov 4, 2014 at 8:14 AM, Shreya Agrawal <s.agra...@gmail.com > wrote: > >> Hey Mitch, >> >> Thanks a lot. This is much more clear now. >> >> I went back and double checked a few things now: >> 1. ODK Aggregate/Form Management/Submission Admin tab only has 2 surveys >> that are there. I will not mark them as complete and leave them as is. >> 2. Briefcase is downloading data for those files that don't have an >> image. So we have 1,743 submissions and only 1,600 images. So that's not >> an issue >> 3. Briefcase does continue to download other data (the new one), but as >> soon as we hit the error for the particular image, it stops (its the same >> unique number ID that keeps coming up). So we have about 151 surveys after >> that particular image that we cannot download. I can see all the files >> that have been submitted on ODK Aggregate, but they are all encrypted. So I >> have no idea which image ODK Briefcase is referring to and how I can >> manually delete it from the Briefcase so that I can still access the 151 >> surveys. >> >> Any thoughts? >> >> Thanks >> >> >> >> >> On Monday, November 3, 2014 2:33:18 PM UTC-5, Mitch Sundt wrote: >>> >>> No. Not having captured a signature would not be the reason for the >>> error. But if you did capture an image, but it was not preserved through >>> the submission process, then that would be a problem. >>> >>> The "Mark as Complete" button is on the ODK Aggregate server, under the >>> Forms Management / Submissions Admin tab; it IS NOT the "Finalize" button >>> on ODK Collect. >>> >>> My comment refers specifically to any submissions displayed under the >>> Forms Management / Submissions Admin tab. If you have submissions >>> displaying on that tab, then they are missing media attachments (e.g., >>> signature files that were captured, but failed to be sent to the server) -- >>> the sending of the submission to the server somehow failed. The user on the >>> device will need to re-send those submissions and their attachments. >>> Choosing "Mark as Complete" will cause problems during the decryption of >>> the affected submission. >>> >>> If your briefcase storage location does have submissions that are >>> missing media attachments (and it sounds like it does), I believe it will >>> affect just those submissions, and not the pulling of data or the >>> generation of the full csv file of all results -- but I have not tested >>> this. Please confirm that this is true through your own testing. I do not >>> know if the affected data will be emitted into the csv or not. I think it >>> is likely that the affected submissions will not have their data written to >>> the csv, but I have not confirmed this. >>> >>> >>> >>> >>> >>> >>> >>> >>> >>> >>> On Sat, Nov 1, 2014 at 12:03 PM, Shreya Agrawal wrote: >>> >>>> Hey Mitch, >>>> >>>> Quick clarification: the signature files don't exist for these surveys >>>> because we never captured them. If the person does not give consent, we >>>> don't have a signature so there are no image files. Therefore, when we are >>>> downloading the data, all these files don't have an image. Is that the >>>> reason for the error? >>>> >>>> If we don't Mark as complete - how do the data collectors upload the >>>> data? These are the files that are finalized? >>>> >>>> I wanted to know how we can still download all these surveys that don't >>>> have an image through the briefcase without the error. Is that possible? >>>> >>>> Also, we have the latest version of ODK Downloaded that's availble on >>>> the Google store. >>>> >>>> Can you please clarify? >>>> >>>> Thanks >>>> Shreya >>>> >>>> -- >>>> -- >>>> Post: opend...@googlegroups.com >>>> Unsubscribe: opendatakit...@googlegroups.com >>>> Options: http://groups.google.com/group/opendatakit?hl=en >>>> >>>> --- >>>> You received this message because you are subscribed to the Google >>>> Groups "ODK Community" group. >>>> To unsubscribe from this group and stop receiving emails from it, send >>>> an email to opendatakit...@googlegroups.com. >>>> For more options, visit https://groups.google.com/d/optout. >>>> >>> >>> >>> >>> -- >>> Mitch Sundt >>> Software Engineer >>> University of Washington >>> mitche...@gmail.com >>> >> -- >> -- >> Post: opend...@googlegroups.com >> Unsubscribe: opendatakit...@googlegroups.com >> Options: http://groups.google.com/group/opendatakit?hl=en >> >> --- >> You received this message because you are subscribed to the Google Groups >> "ODK Community" group. >> To unsubscribe from this group and stop receiving emails from it, send an >> email to opendatakit...@googlegroups.com . >> For more options, visit https://groups.google.com/d/optout. >> > > > > -- > Mitch Sundt > Software Engineer > University of Washington > mitche...@gmail.com >
8

Hey Mitch,

Now we get a new error:

Error decrypting submission uuid9f4f0fe6-b2b4-43ce-91a5-1ffb67a63086 Cause:
org.opendatakit.briefcase.model.FileSystemException: Error decrypting:.temp
Cause: form instance metadata differs from that in manifest
FAILED!

And we can't get access to alot of the data that we have already downloaded
from the Briefcase before.

We are successfully able to Pull ALL the data, but its the exporting
process that is causing the problems. Can we please discuss when you would
be able to work on the ODK Briefcase issue for us? We need to figure out
what's the best way for us to handle the situation on the data collection
end. Should we continue to do data collection or should we stop? We are
afraid that if we continue, we might not be able to get that data.

Thanks so much for your help!

Shreya

··· On Wednesday, November 5, 2014 1:49:31 AM UTC-5, Shreya Agrawal wrote: > > Thanks Mitch! Should we connect via email on this? > > Also, if you fix the Briefcase, will that fix this problem and we can have > access to rest of our data? If you think it will be a reoccurring problem, > we can create a new survey for data collectors to start using so that the > rest of the data does not get corrupted. I am just trying to understand > the severity of the problem. > > Thanks! > > > On Tuesday, November 4, 2014 6:39:16 PM UTC-5, Mitch Sundt wrote: >> >> I will be able to work on this next week, and can send you a new ODK >> Briefcase without this issue (it might omit the problematic survey from the >> csv, but will not cause the csv output to stop). >> >> On Tue, Nov 4, 2014 at 8:14 AM, Shreya Agrawal wrote: >> >>> Hey Mitch, >>> >>> Thanks a lot. This is much more clear now. >>> >>> I went back and double checked a few things now: >>> 1. ODK Aggregate/Form Management/Submission Admin tab only has 2 surveys >>> that are there. I will not mark them as complete and leave them as is. >>> 2. Briefcase is downloading data for those files that don't have an >>> image. So we have 1,743 submissions and only 1,600 images. So that's not >>> an issue >>> 3. Briefcase does continue to download other data (the new one), but as >>> soon as we hit the error for the particular image, it stops (its the same >>> unique number ID that keeps coming up). So we have about 151 surveys after >>> that particular image that we cannot download. I can see all the files >>> that have been submitted on ODK Aggregate, but they are all encrypted. So I >>> have no idea which image ODK Briefcase is referring to and how I can >>> manually delete it from the Briefcase so that I can still access the 151 >>> surveys. >>> >>> Any thoughts? >>> >>> Thanks >>> >>> >>> >>> >>> On Monday, November 3, 2014 2:33:18 PM UTC-5, Mitch Sundt wrote: >>>> >>>> No. Not having captured a signature would not be the reason for the >>>> error. But if you did capture an image, but it was not preserved through >>>> the submission process, then that would be a problem. >>>> >>>> The "Mark as Complete" button is on the ODK Aggregate server, under the >>>> Forms Management / Submissions Admin tab; it IS NOT the "Finalize" button >>>> on ODK Collect. >>>> >>>> My comment refers specifically to any submissions displayed under the >>>> Forms Management / Submissions Admin tab. If you have submissions >>>> displaying on that tab, then they are missing media attachments (e.g., >>>> signature files that were captured, but failed to be sent to the server) -- >>>> the sending of the submission to the server somehow failed. The user on the >>>> device will need to re-send those submissions and their attachments. >>>> Choosing "Mark as Complete" will cause problems during the decryption of >>>> the affected submission. >>>> >>>> If your briefcase storage location does have submissions that are >>>> missing media attachments (and it sounds like it does), I believe it will >>>> affect just those submissions, and not the pulling of data or the >>>> generation of the full csv file of all results -- but I have not tested >>>> this. Please confirm that this is true through your own testing. I do not >>>> know if the affected data will be emitted into the csv or not. I think it >>>> is likely that the affected submissions will not have their data written to >>>> the csv, but I have not confirmed this. >>>> >>>> >>>> >>>> >>>> >>>> >>>> >>>> >>>> >>>> >>>> On Sat, Nov 1, 2014 at 12:03 PM, Shreya Agrawal wrote: >>>> >>>>> Hey Mitch, >>>>> >>>>> Quick clarification: the signature files don't exist for these surveys >>>>> because we never captured them. If the person does not give consent, we >>>>> don't have a signature so there are no image files. Therefore, when we are >>>>> downloading the data, all these files don't have an image. Is that the >>>>> reason for the error? >>>>> >>>>> If we don't Mark as complete - how do the data collectors upload the >>>>> data? These are the files that are finalized? >>>>> >>>>> I wanted to know how we can still download all these surveys that >>>>> don't have an image through the briefcase without the error. Is that >>>>> possible? >>>>> >>>>> Also, we have the latest version of ODK Downloaded that's availble on >>>>> the Google store. >>>>> >>>>> Can you please clarify? >>>>> >>>>> Thanks >>>>> Shreya >>>>> >>>>> -- >>>>> -- >>>>> Post: opend...@googlegroups.com >>>>> Unsubscribe: opendatakit...@googlegroups.com >>>>> Options: http://groups.google.com/group/opendatakit?hl=en >>>>> >>>>> --- >>>>> You received this message because you are subscribed to the Google >>>>> Groups "ODK Community" group. >>>>> To unsubscribe from this group and stop receiving emails from it, send >>>>> an email to opendatakit...@googlegroups.com. >>>>> For more options, visit https://groups.google.com/d/optout. >>>>> >>>> >>>> >>>> >>>> -- >>>> Mitch Sundt >>>> Software Engineer >>>> University of Washington >>>> mitche...@gmail.com >>>> >>> -- >>> -- >>> Post: opend...@googlegroups.com >>> Unsubscribe: opendatakit...@googlegroups.com >>> Options: http://groups.google.com/group/opendatakit?hl=en >>> >>> --- >>> You received this message because you are subscribed to the Google >>> Groups "ODK Community" group. >>> To unsubscribe from this group and stop receiving emails from it, send >>> an email to opendatakit...@googlegroups.com. >>> For more options, visit https://groups.google.com/d/optout. >>> >> >> >> >> -- >> Mitch Sundt >> Software Engineer >> University of Washington >> mitche...@gmail.com >> >
9

Hi

I am getting this same error now decrypting a form. Was there ever any resolution on this issue?

Cause: org.opendatakit.briefcase.model.FileSystemException: Error decrypting:.temp Cause: form instance metadata differs from that in manifest

Cheers,
Jenny

··· On Thursday, 6 November 2014 03:10:16 UTC-8, Shreya Agrawal wrote: > Hey Mitch, > > > Now we get a new error: > > > > Error decrypting submission uuid9f4f0fe6-b2b4-43ce-91a5-1ffb67a63086 Cause: org.opendatakit.briefcase.model.FileSystemException: Error decrypting:.temp Cause: form instance metadata differs from that in manifest > FAILED! > > > And we can't get access to alot of the data that we have already downloaded from the Briefcase before. > > > We are successfully able to Pull ALL the data, but its the exporting process that is causing the problems. Can we please discuss when you would be able to work on the ODK Briefcase issue for us? We need to figure out what's the best way for us to handle the situation on the data collection end. Should we continue to do data collection or should we stop? We are afraid that if we continue, we might not be able to get that data. > > > Thanks so much for your help! > > > Shreya > > > > > On Wednesday, November 5, 2014 1:49:31 AM UTC-5, Shreya Agrawal wrote: > Thanks Mitch! Should we connect via email on this? > > > Also, if you fix the Briefcase, will that fix this problem and we can have access to rest of our data? If you think it will be a reoccurring problem, we can create a new survey for data collectors to start using so that the rest of the data does not get corrupted. I am just trying to understand the severity of the problem. > > > Thanks! > > > On Tuesday, November 4, 2014 6:39:16 PM UTC-5, Mitch Sundt wrote: > I will be able to work on this next week, and can send you a new ODK Briefcase without this issue (it might omit the problematic survey from the csv, but will not cause the csv output to stop). > > > > On Tue, Nov 4, 2014 at 8:14 AM, Shreya Agrawal wrote: > > Hey Mitch, > > > Thanks a lot. This is much more clear now. > > > I went back and double checked a few things now: > 1. ODK Aggregate/Form Management/Submission Admin tab only has 2 surveys that are there. I will not mark them as complete and leave them as is. > 2. Briefcase is downloading data for those files that don't have an image. So we have 1,743 submissions and only 1,600 images. So that's not an issue > 3. Briefcase does continue to download other data (the new one), but as soon as we hit the error for the particular image, it stops (its the same unique number ID that keeps coming up). So we have about 151 surveys after that particular image that we cannot download. I can see all the files that have been submitted on ODK Aggregate, but they are all encrypted. So I have no idea which image ODK Briefcase is referring to and how I can manually delete it from the Briefcase so that I can still access the 151 surveys. > > > Any thoughts? > > > Thanks > > > > > > > On Monday, November 3, 2014 2:33:18 PM UTC-5, Mitch Sundt wrote: > > No. Not having captured a signature would not be the reason for the error. But if you did capture an image, but it was not preserved through the submission process, then that would be a problem. > > > The "Mark as Complete" button is on the ODK Aggregate server, under the Forms Management / Submissions Admin tab; it IS NOT the "Finalize" button on ODK Collect. > > > My comment refers specifically to any submissions displayed under the Forms Management / Submissions Admin tab. If you have submissions displaying on that tab, then they are missing media attachments (e.g., signature files that were captured, but failed to be sent to the server) -- the sending of the submission to the server somehow failed. The user on the device will need to re-send those submissions and their attachments. Choosing "Mark as Complete" will cause problems during the decryption of the affected submission. > > If your briefcase storage location does have submissions that are missing media attachments (and it sounds like it does), I believe it will affect just those submissions, and not the pulling of data or the generation of the full csv file of all results -- but I have not tested this. Please confirm that this is true through your own testing. I do not know if the affected data will be emitted into the csv or not. I think it is likely that the affected submissions will not have their data written to the csv, but I have not confirmed this. > > > > > > > > > > > > > > > > > On Sat, Nov 1, 2014 at 12:03 PM, Shreya Agrawal wrote: > Hey Mitch, > > > > Quick clarification: the signature files don't exist for these surveys because we never captured them. If the person does not give consent, we don't have a signature so there are no image files. Therefore, when we are downloading the data, all these files don't have an image. Is that the reason for the error? > > > > If we don't Mark as complete - how do the data collectors upload the data? These are the files that are finalized? > > > > I wanted to know how we can still download all these surveys that don't have an image through the briefcase without the error. Is that possible? > > > > Also, we have the latest version of ODK Downloaded that's availble on the Google store. > > > > Can you please clarify? > > > > > > Thanks > > Shreya > > > > -- > > -- > > Post: opend...@googlegroups.com > > Unsubscribe: opendatakit...@googlegroups.com > > Options: http://groups.google.com/group/opendatakit?hl=en > > > > --- > > You received this message because you are subscribed to the Google Groups "ODK Community" group. > > To unsubscribe from this group and stop receiving emails from it, send an email to opendatakit...@googlegroups.com. > > For more options, visit https://groups.google.com/d/optout. > > > > > -- > > Mitch Sundt > Software Engineer > University of Washington > mitche...@gmail.com > > > > > > > > > -- > > -- > > Post: opend...@googlegroups.com > > Unsubscribe: opendatakit...@googlegroups.com > > Options: http://groups.google.com/group/opendatakit?hl=en > > > > --- > > You received this message because you are subscribed to the Google Groups "ODK Community" group. > > To unsubscribe from this group and stop receiving emails from it, send an email to opendatakit...@googlegroups.com. > > For more options, visit https://groups.google.com/d/optout. > > > > > -- > > Mitch Sundt > Software Engineer > University of Washington > mitche...@gmail.com
10

That issue was resolved in the 1.4.5 ODK Aggregate and 1.4.5 ODK Briefcase
releases (you must upgrade to both).
It relates to using the Mark-as-Complete action on the server when using
encrypted forms.

Mitch

··· On Wed, Mar 11, 2015 at 5:05 PM, wrote:

Hi

I am getting this same error now decrypting a form. Was there ever any
resolution on this issue?

Cause: org.opendatakit.briefcase.model.FileSystemException: Error
decrypting:.temp Cause: form instance metadata differs from that in manifest

Cheers,
Jenny

On Thursday, 6 November 2014 03:10:16 UTC-8, Shreya Agrawal wrote:

Hey Mitch,

Now we get a new error:

Error decrypting submission uuid9f4f0fe6-b2b4-43ce-91a5-1ffb67a63086
Cause: org.opendatakit.briefcase.model.FileSystemException: Error
decrypting:.temp Cause: form instance metadata differs from that in manifest
FAILED!

And we can't get access to alot of the data that we have already
downloaded from the Briefcase before.

We are successfully able to Pull ALL the data, but its the exporting
process that is causing the problems. Can we please discuss when you would
be able to work on the ODK Briefcase issue for us? We need to figure out
what's the best way for us to handle the situation on the data collection
end. Should we continue to do data collection or should we stop? We are
afraid that if we continue, we might not be able to get that data.

Thanks so much for your help!

Shreya

On Wednesday, November 5, 2014 1:49:31 AM UTC-5, Shreya Agrawal wrote:
Thanks Mitch! Should we connect via email on this?

Also, if you fix the Briefcase, will that fix this problem and we can
have access to rest of our data? If you think it will be a reoccurring
problem, we can create a new survey for data collectors to start using so
that the rest of the data does not get corrupted. I am just trying to
understand the severity of the problem.

Thanks!

On Tuesday, November 4, 2014 6:39:16 PM UTC-5, Mitch Sundt wrote:
I will be able to work on this next week, and can send you a new ODK
Briefcase without this issue (it might omit the problematic survey from the
csv, but will not cause the csv output to stop).

On Tue, Nov 4, 2014 at 8:14 AM, Shreya Agrawal s.agra...@gmail.com wrote:

Hey Mitch,

Thanks a lot. This is much more clear now.

I went back and double checked a few things now:

  1. ODK Aggregate/Form Management/Submission Admin tab only has 2 surveys
    that are there. I will not mark them as complete and leave them as is.
  2. Briefcase is downloading data for those files that don't have an
    image. So we have 1,743 submissions and only 1,600 images. So that's not
    an issue
  3. Briefcase does continue to download other data (the new one), but as
    soon as we hit the error for the particular image, it stops (its the same
    unique number ID that keeps coming up). So we have about 151 surveys after
    that particular image that we cannot download. I can see all the files
    that have been submitted on ODK Aggregate, but they are all encrypted. So I
    have no idea which image ODK Briefcase is referring to and how I can
    manually delete it from the Briefcase so that I can still access the 151
    surveys.

Any thoughts?

Thanks

On Monday, November 3, 2014 2:33:18 PM UTC-5, Mitch Sundt wrote:

No. Not having captured a signature would not be the reason for the
error. But if you did capture an image, but it was not preserved through
the submission process, then that would be a problem.

The "Mark as Complete" button is on the ODK Aggregate server, under the
Forms Management / Submissions Admin tab; it IS NOT the "Finalize" button
on ODK Collect.

My comment refers specifically to any submissions displayed under the
Forms Management / Submissions Admin tab. If you have submissions
displaying on that tab, then they are missing media attachments (e.g.,
signature files that were captured, but failed to be sent to the server) --
the sending of the submission to the server somehow failed. The user on the
device will need to re-send those submissions and their attachments.
Choosing "Mark as Complete" will cause problems during the decryption of
the affected submission.

If your briefcase storage location does have submissions that are
missing media attachments (and it sounds like it does), I believe it will
affect just those submissions, and not the pulling of data or the
generation of the full csv file of all results -- but I have not tested
this. Please confirm that this is true through your own testing. I do not
know if the affected data will be emitted into the csv or not. I think it
is likely that the affected submissions will not have their data written to
the csv, but I have not confirmed this.

On Sat, Nov 1, 2014 at 12:03 PM, Shreya Agrawal s.agra...@gmail.com wrote:
Hey Mitch,

Quick clarification: the signature files don't exist for these surveys
because we never captured them. If the person does not give consent, we
don't have a signature so there are no image files. Therefore, when we are
downloading the data, all these files don't have an image. Is that the
reason for the error?

If we don't Mark as complete - how do the data collectors upload the
data? These are the files that are finalized?

I wanted to know how we can still download all these surveys that don't
have an image through the briefcase without the error. Is that possible?

Also, we have the latest version of ODK Downloaded that's availble on
the Google store.

Can you please clarify?

Thanks

Shreya

--

--

Post: opend...@googlegroups.com

Unsubscribe: opendatakit...@googlegroups.com

Options: http://groups.google.com/group/opendatakit?hl=en

You received this message because you are subscribed to the Google
Groups "ODK Community" group.

To unsubscribe from this group and stop receiving emails from it, send
an email to opendatakit...@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--

Mitch Sundt
Software Engineer
University of Washington
mitche...@gmail.com

--

--

Post: opend...@googlegroups.com

Unsubscribe: opendatakit...@googlegroups.com

Options: http://groups.google.com/group/opendatakit?hl=en

You received this message because you are subscribed to the Google
Groups "ODK Community" group.

To unsubscribe from this group and stop receiving emails from it, send
an email to opendatakit...@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--

Mitch Sundt
Software Engineer
University of Washington
mitche...@gmail.com

--

Post: opendatakit@googlegroups.com
Unsubscribe: opendatakit+unsubscribe@googlegroups.com
Options: http://groups.google.com/group/opendatakit?hl=en

You received this message because you are subscribed to the Google Groups
"ODK Community" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to opendatakit+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--
Mitch Sundt
Software Engineer
University of Washington
mitchellsundt@gmail.com

11

Thanks for the quick response. I am already using those versions and have never used the Mark-as-Complete action on the server. In fact, I have never noticed that action.

The error seems to be associated with one instance, but there are a number that are on the server and not decrypted after this failure. Any other thoughts?

Starting Export...
Processing instance: uuid071ebb87-435a-44b7-b251-f528e58951b0
Processing instance: uuid101cfa87-7abd-473d-b56a-6baaa251d73d
Processing instance: uuid1b54a30b-b221-4134-8737-f77e6dcd2e2f
Processing instance: uuid2289894b-5aaf-45ca-8c55-98b4fe4e923c
Processing instance: uuid23524cb1-2ca2-4cf0-8333-a53499f18403
Processing instance: uuid35235650-cd0b-43a0-b4ff-0d98dca480ff
Error decrypting submission uuid35235650-cd0b-43a0-b4ff-0d98dca480ff Cause: org.opendatakit.briefcase.model.FileSystemException: Error decrypting:.temp Cause: form instance metadata differs from that in manifest
FAILED!

··· On Wednesday, 11 March 2015 18:07:09 UTC-7, Mitch Sundt wrote: > That issue was resolved in the 1.4.5 ODK Aggregate and 1.4.5 ODK Briefcase releases (you must upgrade to both). > > It relates to using the Mark-as-Complete action on the server when using encrypted forms. > > Mitch > > > > > > > On Wed, Mar 11, 2015 at 5:05 PM, wrote: > Hi > > > > I am getting this same error now decrypting a form. Was there ever any resolution on this issue? > > > > Cause: org.opendatakit.briefcase.model.FileSystemException: Error decrypting:.temp Cause: form instance metadata differs from that in manifest > > > > Cheers, > > Jenny > > > > > > On Thursday, 6 November 2014 03:10:16 UTC-8, Shreya Agrawal wrote: > > > Hey Mitch, > > > > > > > > > Now we get a new error: > > > > > > > > > > > > Error decrypting submission uuid9f4f0fe6-b2b4-43ce-91a5-1ffb67a63086 Cause: org.opendatakit.briefcase.model.FileSystemException: Error decrypting:.temp Cause: form instance metadata differs from that in manifest > > > FAILED! > > > > > > > > > And we can't get access to alot of the data that we have already downloaded from the Briefcase before. > > > > > > > > > We are successfully able to Pull ALL the data, but its the exporting process that is causing the problems. Can we please discuss when you would be able to work on the ODK Briefcase issue for us? We need to figure out what's the best way for us to handle the situation on the data collection end. Should we continue to do data collection or should we stop? We are afraid that if we continue, we might not be able to get that data. > > > > > > > > > Thanks so much for your help! > > > > > > > > > Shreya > > > > > > > > > > > > > > > On Wednesday, November 5, 2014 1:49:31 AM UTC-5, Shreya Agrawal wrote: > > > Thanks Mitch! Should we connect via email on this? > > > > > > > > > Also, if you fix the Briefcase, will that fix this problem and we can have access to rest of our data? If you think it will be a reoccurring problem, we can create a new survey for data collectors to start using so that the rest of the data does not get corrupted. I am just trying to understand the severity of the problem. > > > > > > > > > Thanks! > > > > > > > > > On Tuesday, November 4, 2014 6:39:16 PM UTC-5, Mitch Sundt wrote: > > > I will be able to work on this next week, and can send you a new ODK Briefcase without this issue (it might omit the problematic survey from the csv, but will not cause the csv output to stop). > > > > > > > > > > > > On Tue, Nov 4, 2014 at 8:14 AM, Shreya Agrawal wrote: > > > > > > Hey Mitch, > > > > > > > > > Thanks a lot. This is much more clear now. > > > > > > > > > I went back and double checked a few things now: > > > 1. ODK Aggregate/Form Management/Submission Admin tab only has 2 surveys that are there. I will not mark them as complete and leave them as is. > > > 2. Briefcase is downloading data for those files that don't have an image. So we have 1,743 submissions and only 1,600 images. So that's not an issue > > > 3. Briefcase does continue to download other data (the new one), but as soon as we hit the error for the particular image, it stops (its the same unique number ID that keeps coming up). So we have about 151 surveys after that particular image that we cannot download. I can see all the files that have been submitted on ODK Aggregate, but they are all encrypted. So I have no idea which image ODK Briefcase is referring to and how I can manually delete it from the Briefcase so that I can still access the 151 surveys. > > > > > > > > > Any thoughts? > > > > > > > > > Thanks > > > > > > > > > > > > > > > > > > > > > On Monday, November 3, 2014 2:33:18 PM UTC-5, Mitch Sundt wrote: > > > > > > No. Not having captured a signature would not be the reason for the error. But if you did capture an image, but it was not preserved through the submission process, then that would be a problem. > > > > > > > > > The "Mark as Complete" button is on the ODK Aggregate server, under the Forms Management / Submissions Admin tab; it IS NOT the "Finalize" button on ODK Collect. > > > > > > > > > My comment refers specifically to any submissions displayed under the Forms Management / Submissions Admin tab. If you have submissions displaying on that tab, then they are missing media attachments (e.g., signature files that were captured, but failed to be sent to the server) -- the sending of the submission to the server somehow failed. The user on the device will need to re-send those submissions and their attachments. Choosing "Mark as Complete" will cause problems during the decryption of the affected submission. > > > > > > If your briefcase storage location does have submissions that are missing media attachments (and it sounds like it does), I believe it will affect just those submissions, and not the pulling of data or the generation of the full csv file of all results -- but I have not tested this. Please confirm that this is true through your own testing. I do not know if the affected data will be emitted into the csv or not. I think it is likely that the affected submissions will not have their data written to the csv, but I have not confirmed this. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > On Sat, Nov 1, 2014 at 12:03 PM, Shreya Agrawal wrote: > > > Hey Mitch, > > > > > > > > > > > > Quick clarification: the signature files don't exist for these surveys because we never captured them. If the person does not give consent, we don't have a signature so there are no image files. Therefore, when we are downloading the data, all these files don't have an image. Is that the reason for the error? > > > > > > > > > > > > If we don't Mark as complete - how do the data collectors upload the data? These are the files that are finalized? > > > > > > > > > > > > I wanted to know how we can still download all these surveys that don't have an image through the briefcase without the error. Is that possible? > > > > > > > > > > > > Also, we have the latest version of ODK Downloaded that's availble on the Google store. > > > > > > > > > > > > Can you please clarify? > > > > > > > > > > > > > > > > > > Thanks > > > > > > Shreya > > > > > > > > > > > > -- > > > > > > -- > > > > > > Post: opend...@googlegroups.com > > > > > > Unsubscribe: opendatakit...@googlegroups.com > > > > > > Options: http://groups.google.com/group/opendatakit?hl=en > > > > > > > > > > > > --- > > > > > > You received this message because you are subscribed to the Google Groups "ODK Community" group. > > > > > > To unsubscribe from this group and stop receiving emails from it, send an email to opendatakit...@googlegroups.com. > > > > > > For more options, visit https://groups.google.com/d/optout. > > > > > > > > > > > > > > > -- > > > > > > Mitch Sundt > > > Software Engineer > > > University of Washington > > > mitche...@gmail.com > > > > > > > > > > > > > > > > > > > > > > > > > > > -- > > > > > > -- > > > > > > Post: opend...@googlegroups.com > > > > > > Unsubscribe: opendatakit...@googlegroups.com > > > > > > Options: http://groups.google.com/group/opendatakit?hl=en > > > > > > > > > > > > --- > > > > > > You received this message because you are subscribed to the Google Groups "ODK Community" group. > > > > > > To unsubscribe from this group and stop receiving emails from it, send an email to opendatakit...@googlegroups.com. > > > > > > For more options, visit https://groups.google.com/d/optout. > > > > > > > > > > > > > > > -- > > > > > > Mitch Sundt > > > Software Engineer > > > University of Washington > > > mitche...@gmail.com > > > > -- > > -- > > Post: opend...@googlegroups.com > > Unsubscribe: opendatakit...@googlegroups.com > > Options: http://groups.google.com/group/opendatakit?hl=en > > > > --- > > You received this message because you are subscribed to the Google Groups "ODK Community" group. > > To unsubscribe from this group and stop receiving emails from it, send an email to opendatakit...@googlegroups.com. > > For more options, visit https://groups.google.com/d/optout. > > > > > -- > > Mitch Sundt > Software Engineer > University of Washington > mitche...@gmail.com
12

Hmm.

The code:

··· ----------------------------- boolean same = decryptedSubmission.formId.equals(encryptionManifest.formId); 
if ( !same ) {
  throw new FileSystemException("Error decrypting:" +

unencryptedDir.getName()
+ " Cause: form instance metadata differs from that in manifest");
}

Which is only possible if there is something very bizarre going on -- this
indicates that the formId identified in the encrypted-submission manifest
does not match the formId within the decrypted submission body. That
should be impossible.

Much as I would like to, I have no way of investigating this error without
compromising your data set and private encryption key.

The work-around is to discard this problematic submission from your result
set by navigating into the ODK Briefcase Storage directory (this is
generally located in your user directory. E.g., for Windows, if you open a
file browser, click on Downloads, then navigate up one directory, you will
be in your user directory and should see an ODK Briefcase Storage directory
there.

Within that directory, you would navigate into forms/YOUR_FORM_ID/instances/

Simply move the uuid35235650-cd0b-43a0-b4ff-0d98dca480ff directory out of
the ODK Briefcase Storage directory.

You should then be able to decrypt and export the remaining surveys.

On Wed, Mar 11, 2015 at 6:27 PM, jennifer.linnea@gmail.com wrote:

Thanks for the quick response. I am already using those versions and have
never used the Mark-as-Complete action on the server. In fact, I have never
noticed that action.

The error seems to be associated with one instance, but there are a number
that are on the server and not decrypted after this failure. Any other
thoughts?

Starting Export...
Processing instance: uuid071ebb87-435a-44b7-b251-f528e58951b0
Processing instance: uuid101cfa87-7abd-473d-b56a-6baaa251d73d
Processing instance: uuid1b54a30b-b221-4134-8737-f77e6dcd2e2f
Processing instance: uuid2289894b-5aaf-45ca-8c55-98b4fe4e923c
Processing instance: uuid23524cb1-2ca2-4cf0-8333-a53499f18403
Processing instance: uuid35235650-cd0b-43a0-b4ff-0d98dca480ff
Error decrypting submission uuid35235650-cd0b-43a0-b4ff-0d98dca480ff
Cause: org.opendatakit.briefcase.model.FileSystemException: Error
decrypting:.temp Cause: form instance metadata differs from that in manifest
FAILED!

On Wednesday, 11 March 2015 18:07:09 UTC-7, Mitch Sundt wrote:

That issue was resolved in the 1.4.5 ODK Aggregate and 1.4.5 ODK
Briefcase releases (you must upgrade to both).

It relates to using the Mark-as-Complete action on the server when using
encrypted forms.

Mitch

On Wed, Mar 11, 2015 at 5:05 PM, jennife...@gmail.com wrote:
Hi

I am getting this same error now decrypting a form. Was there ever any
resolution on this issue?

Cause: org.opendatakit.briefcase.model.FileSystemException: Error
decrypting:.temp Cause: form instance metadata differs from that in manifest

Cheers,

Jenny

On Thursday, 6 November 2014 03:10:16 UTC-8, Shreya Agrawal wrote:

Hey Mitch,

Now we get a new error:

Error decrypting submission uuid9f4f0fe6-b2b4-43ce-91a5-1ffb67a63086
Cause: org.opendatakit.briefcase.model.FileSystemException: Error
decrypting:.temp Cause: form instance metadata differs from that in manifest

FAILED!

And we can't get access to alot of the data that we have already
downloaded from the Briefcase before.

We are successfully able to Pull ALL the data, but its the exporting
process that is causing the problems. Can we please discuss when you would
be able to work on the ODK Briefcase issue for us? We need to figure out
what's the best way for us to handle the situation on the data collection
end. Should we continue to do data collection or should we stop? We are
afraid that if we continue, we might not be able to get that data.

Thanks so much for your help!

Shreya

On Wednesday, November 5, 2014 1:49:31 AM UTC-5, Shreya Agrawal wrote:

Thanks Mitch! Should we connect via email on this?

Also, if you fix the Briefcase, will that fix this problem and we can
have access to rest of our data? If you think it will be a reoccurring
problem, we can create a new survey for data collectors to start using so
that the rest of the data does not get corrupted. I am just trying to
understand the severity of the problem.

Thanks!

On Tuesday, November 4, 2014 6:39:16 PM UTC-5, Mitch Sundt wrote:

I will be able to work on this next week, and can send you a new ODK
Briefcase without this issue (it might omit the problematic survey from the
csv, but will not cause the csv output to stop).

On Tue, Nov 4, 2014 at 8:14 AM, Shreya Agrawal s.agra...@gmail.com wrote:

Hey Mitch,

Thanks a lot. This is much more clear now.

I went back and double checked a few things now:

  1. ODK Aggregate/Form Management/Submission Admin tab only has 2
    surveys that are there. I will not mark them as complete and leave them as
    is.
  1. Briefcase is downloading data for those files that don't have an
    image. So we have 1,743 submissions and only 1,600 images. So that's not
    an issue
  1. Briefcase does continue to download other data (the new one), but
    as soon as we hit the error for the particular image, it stops (its the
    same unique number ID that keeps coming up). So we have about 151 surveys
    after that particular image that we cannot download. I can see all the
    files that have been submitted on ODK Aggregate, but they are all
    encrypted. So I have no idea which image ODK Briefcase is referring to and
    how I can manually delete it from the Briefcase so that I can still access
    the 151 surveys.

Any thoughts?

Thanks

On Monday, November 3, 2014 2:33:18 PM UTC-5, Mitch Sundt wrote:

No. Not having captured a signature would not be the reason for the
error. But if you did capture an image, but it was not preserved through
the submission process, then that would be a problem.

The "Mark as Complete" button is on the ODK Aggregate server, under
the Forms Management / Submissions Admin tab; it IS NOT the "Finalize"
button on ODK Collect.

My comment refers specifically to any submissions displayed under the
Forms Management / Submissions Admin tab. If you have submissions
displaying on that tab, then they are missing media attachments (e.g.,
signature files that were captured, but failed to be sent to the server) --
the sending of the submission to the server somehow failed. The user on the
device will need to re-send those submissions and their attachments.
Choosing "Mark as Complete" will cause problems during the decryption of
the affected submission.

If your briefcase storage location does have submissions that are
missing media attachments (and it sounds like it does), I believe it will
affect just those submissions, and not the pulling of data or the
generation of the full csv file of all results -- but I have not tested
this. Please confirm that this is true through your own testing. I do not
know if the affected data will be emitted into the csv or not. I think it
is likely that the affected submissions will not have their data written to
the csv, but I have not confirmed this.

On Sat, Nov 1, 2014 at 12:03 PM, Shreya Agrawal s.agra...@gmail.com wrote:

Hey Mitch,

Quick clarification: the signature files don't exist for these surveys
because we never captured them. If the person does not give consent, we
don't have a signature so there are no image files. Therefore, when we are
downloading the data, all these files don't have an image. Is that the
reason for the error?

If we don't Mark as complete - how do the data collectors upload the
data? These are the files that are finalized?

I wanted to know how we can still download all these surveys that
don't have an image through the briefcase without the error. Is that
possible?

Also, we have the latest version of ODK Downloaded that's availble on
the Google store.

Can you please clarify?

Thanks

Shreya

--

--

Post: opend...@googlegroups.com

Unsubscribe: opendatakit...@googlegroups.com

Options: http://groups.google.com/group/opendatakit?hl=en

You received this message because you are subscribed to the Google
Groups "ODK Community" group.

To unsubscribe from this group and stop receiving emails from it, send
an email to opendatakit...@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--

Mitch Sundt

Software Engineer

University of Washington

mitche...@gmail.com

--

--

Post: opend...@googlegroups.com

Unsubscribe: opendatakit...@googlegroups.com

Options: http://groups.google.com/group/opendatakit?hl=en

You received this message because you are subscribed to the Google
Groups "ODK Community" group.

To unsubscribe from this group and stop receiving emails from it, send
an email to opendatakit...@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--

Mitch Sundt

Software Engineer

University of Washington

mitche...@gmail.com

--

--

Post: opend...@googlegroups.com

Unsubscribe: opendatakit...@googlegroups.com

Options: http://groups.google.com/group/opendatakit?hl=en

You received this message because you are subscribed to the Google
Groups "ODK Community" group.

To unsubscribe from this group and stop receiving emails from it, send
an email to opendatakit...@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--

Mitch Sundt
Software Engineer
University of Washington
mitche...@gmail.com

--

Post: opendatakit@googlegroups.com
Unsubscribe: opendatakit+unsubscribe@googlegroups.com
Options: http://groups.google.com/group/opendatakit?hl=en

You received this message because you are subscribed to the Google Groups
"ODK Community" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to opendatakit+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--
Mitch Sundt
Software Engineer
University of Washington
mitchellsundt@gmail.com

13

Thanks Mitch,

That has worked, so we only have that one record missing from the database
now.

It is now sounding like we may not even need the encryption, which we had
done as a precaution. It may be possible to share access with you if you
think there might be a way to recover that one record. I am just a little
nervous of keeping the encryption if this is a problem with the form or
tablets.

Cheers,
Jenny

··· On Thu, Mar 12, 2015 at 10:41 AM, Mitch Sundt wrote:

Hmm.

The code:

boolean same =

decryptedSubmission.formId.equals(encryptionManifest.formId);

if ( !same ) {
  throw new FileSystemException("Error decrypting:" +

unencryptedDir.getName()
+ " Cause: form instance metadata differs from that in
manifest");
}

Which is only possible if there is something very bizarre going on -- this
indicates that the formId identified in the encrypted-submission manifest
does not match the formId within the decrypted submission body. That
should be impossible.

Much as I would like to, I have no way of investigating this error without
compromising your data set and private encryption key.

The work-around is to discard this problematic submission from your result
set by navigating into the ODK Briefcase Storage directory (this is
generally located in your user directory. E.g., for Windows, if you open a
file browser, click on Downloads, then navigate up one directory, you will
be in your user directory and should see an ODK Briefcase Storage directory
there.

Within that directory, you would navigate into
forms/YOUR_FORM_ID/instances/

Simply move the uuid35235650-cd0b-43a0-b4ff-0d98dca480ff directory out of
the ODK Briefcase Storage directory.

You should then be able to decrypt and export the remaining surveys.

On Wed, Mar 11, 2015 at 6:27 PM, jennifer.linnea@gmail.com wrote:

Thanks for the quick response. I am already using those versions and have
never used the Mark-as-Complete action on the server. In fact, I have never
noticed that action.

The error seems to be associated with one instance, but there are a
number that are on the server and not decrypted after this failure. Any
other thoughts?

Starting Export...
Processing instance: uuid071ebb87-435a-44b7-b251-f528e58951b0
Processing instance: uuid101cfa87-7abd-473d-b56a-6baaa251d73d
Processing instance: uuid1b54a30b-b221-4134-8737-f77e6dcd2e2f
Processing instance: uuid2289894b-5aaf-45ca-8c55-98b4fe4e923c
Processing instance: uuid23524cb1-2ca2-4cf0-8333-a53499f18403
Processing instance: uuid35235650-cd0b-43a0-b4ff-0d98dca480ff
Error decrypting submission uuid35235650-cd0b-43a0-b4ff-0d98dca480ff
Cause: org.opendatakit.briefcase.model.FileSystemException: Error
decrypting:.temp Cause: form instance metadata differs from that in manifest
FAILED!

On Wednesday, 11 March 2015 18:07:09 UTC-7, Mitch Sundt wrote:

That issue was resolved in the 1.4.5 ODK Aggregate and 1.4.5 ODK
Briefcase releases (you must upgrade to both).

It relates to using the Mark-as-Complete action on the server when
using encrypted forms.

Mitch

On Wed, Mar 11, 2015 at 5:05 PM, jennife...@gmail.com wrote:
Hi

I am getting this same error now decrypting a form. Was there ever any
resolution on this issue?

Cause: org.opendatakit.briefcase.model.FileSystemException: Error
decrypting:.temp Cause: form instance metadata differs from that in manifest

Cheers,

Jenny

On Thursday, 6 November 2014 03:10:16 UTC-8, Shreya Agrawal wrote:

Hey Mitch,

Now we get a new error:

Error decrypting submission uuid9f4f0fe6-b2b4-43ce-91a5-1ffb67a63086
Cause: org.opendatakit.briefcase.model.FileSystemException: Error
decrypting:.temp Cause: form instance metadata differs from that in manifest

FAILED!

And we can't get access to alot of the data that we have already
downloaded from the Briefcase before.

We are successfully able to Pull ALL the data, but its the exporting
process that is causing the problems. Can we please discuss when you would
be able to work on the ODK Briefcase issue for us? We need to figure out
what's the best way for us to handle the situation on the data collection
end. Should we continue to do data collection or should we stop? We are
afraid that if we continue, we might not be able to get that data.

Thanks so much for your help!

Shreya

On Wednesday, November 5, 2014 1:49:31 AM UTC-5, Shreya Agrawal wrote:

Thanks Mitch! Should we connect via email on this?

Also, if you fix the Briefcase, will that fix this problem and we can
have access to rest of our data? If you think it will be a reoccurring
problem, we can create a new survey for data collectors to start using so
that the rest of the data does not get corrupted. I am just trying to
understand the severity of the problem.

Thanks!

On Tuesday, November 4, 2014 6:39:16 PM UTC-5, Mitch Sundt wrote:

I will be able to work on this next week, and can send you a new ODK
Briefcase without this issue (it might omit the problematic survey from the
csv, but will not cause the csv output to stop).

On Tue, Nov 4, 2014 at 8:14 AM, Shreya Agrawal s.agra...@gmail.com wrote:

Hey Mitch,

Thanks a lot. This is much more clear now.

I went back and double checked a few things now:

  1. ODK Aggregate/Form Management/Submission Admin tab only has 2
    surveys that are there. I will not mark them as complete and leave them as
    is.
  1. Briefcase is downloading data for those files that don't have an
    image. So we have 1,743 submissions and only 1,600 images. So that's not
    an issue
  1. Briefcase does continue to download other data (the new one), but
    as soon as we hit the error for the particular image, it stops (its the
    same unique number ID that keeps coming up). So we have about 151 surveys
    after that particular image that we cannot download. I can see all the
    files that have been submitted on ODK Aggregate, but they are all
    encrypted. So I have no idea which image ODK Briefcase is referring to and
    how I can manually delete it from the Briefcase so that I can still access
    the 151 surveys.

Any thoughts?

Thanks

On Monday, November 3, 2014 2:33:18 PM UTC-5, Mitch Sundt wrote:

No. Not having captured a signature would not be the reason for the
error. But if you did capture an image, but it was not preserved through
the submission process, then that would be a problem.

The "Mark as Complete" button is on the ODK Aggregate server, under
the Forms Management / Submissions Admin tab; it IS NOT the "Finalize"
button on ODK Collect.

My comment refers specifically to any submissions displayed under the
Forms Management / Submissions Admin tab. If you have submissions
displaying on that tab, then they are missing media attachments (e.g.,
signature files that were captured, but failed to be sent to the server) --
the sending of the submission to the server somehow failed. The user on the
device will need to re-send those submissions and their attachments.
Choosing "Mark as Complete" will cause problems during the decryption of
the affected submission.

If your briefcase storage location does have submissions that are
missing media attachments (and it sounds like it does), I believe it will
affect just those submissions, and not the pulling of data or the
generation of the full csv file of all results -- but I have not tested
this. Please confirm that this is true through your own testing. I do not
know if the affected data will be emitted into the csv or not. I think it
is likely that the affected submissions will not have their data written to
the csv, but I have not confirmed this.

On Sat, Nov 1, 2014 at 12:03 PM, Shreya Agrawal s.agra...@gmail.com wrote:

Hey Mitch,

Quick clarification: the signature files don't exist for these
surveys because we never captured them. If the person does not give
consent, we don't have a signature so there are no image files. Therefore,
when we are downloading the data, all these files don't have an image. Is
that the reason for the error?

If we don't Mark as complete - how do the data collectors upload the
data? These are the files that are finalized?

I wanted to know how we can still download all these surveys that
don't have an image through the briefcase without the error. Is that
possible?

Also, we have the latest version of ODK Downloaded that's availble on
the Google store.

Can you please clarify?

Thanks

Shreya

--

--

Post: opend...@googlegroups.com

Unsubscribe: opendatakit...@googlegroups.com

Options: http://groups.google.com/group/opendatakit?hl=en

You received this message because you are subscribed to the Google
Groups "ODK Community" group.

To unsubscribe from this group and stop receiving emails from it,
send an email to opendatakit...@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--

Mitch Sundt

Software Engineer

University of Washington

mitche...@gmail.com

--

--

Post: opend...@googlegroups.com

Unsubscribe: opendatakit...@googlegroups.com

Options: http://groups.google.com/group/opendatakit?hl=en

You received this message because you are subscribed to the Google
Groups "ODK Community" group.

To unsubscribe from this group and stop receiving emails from it,
send an email to opendatakit...@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--

Mitch Sundt

Software Engineer

University of Washington

mitche...@gmail.com

--

--

Post: opend...@googlegroups.com

Unsubscribe: opendatakit...@googlegroups.com

Options: http://groups.google.com/group/opendatakit?hl=en

You received this message because you are subscribed to the Google
Groups "ODK Community" group.

To unsubscribe from this group and stop receiving emails from it, send
an email to opendatakit...@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--

Mitch Sundt
Software Engineer
University of Washington
mitche...@gmail.com

--

Post: opendatakit@googlegroups.com
Unsubscribe: opendatakit+unsubscribe@googlegroups.com
Options: http://groups.google.com/group/opendatakit?hl=en

You received this message because you are subscribed to the Google Groups
"ODK Community" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to opendatakit+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--
Mitch Sundt
Software Engineer
University of Washington
mitchellsundt@gmail.com

--

Post: opendatakit@googlegroups.com
Unsubscribe: opendatakit+unsubscribe@googlegroups.com
Options: http://groups.google.com/group/opendatakit?hl=en

You received this message because you are subscribed to a topic in the
Google Groups "ODK Community" group.
To unsubscribe from this topic, visit
https://groups.google.com/d/topic/opendatakit/IE7--7ys85o/unsubscribe.
To unsubscribe from this group and all its topics, send an email to
opendatakit+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

14

On the plus side, this is the first time I have ever heard of anyone
getting this error. And a number of sites are actively using encrypted
surveys.

If you are willing, you can copy your entire ODK Briefcase Storage
directory, then delete all the forms except this problematic one, and,
within this problematic form, delete all instances except the one that was
causing problems.

Then zip up that slimmed-down directory and your private key and send it
directly to me ( mitchellsundt@gmail.com ).

It is unlikely that I will be able to resolve this issue anytime soon, as,
even with this information, I suspect it won't be enough to indicate a root
cause for the problem.

··· On Thu, Mar 12, 2015 at 10:55 AM, Jenny Smith wrote:

Thanks Mitch,

That has worked, so we only have that one record missing from the database
now.

It is now sounding like we may not even need the encryption, which we had
done as a precaution. It may be possible to share access with you if you
think there might be a way to recover that one record. I am just a little
nervous of keeping the encryption if this is a problem with the form or
tablets.

Cheers,
Jenny

On Thu, Mar 12, 2015 at 10:41 AM, Mitch Sundt mitchellsundt@gmail.com wrote:

Hmm.

The code:

boolean same =

decryptedSubmission.formId.equals(encryptionManifest.formId);

if ( !same ) {
  throw new FileSystemException("Error decrypting:" +

unencryptedDir.getName()
+ " Cause: form instance metadata differs from that in
manifest");
}

Which is only possible if there is something very bizarre going on --
this indicates that the formId identified in the encrypted-submission
manifest does not match the formId within the decrypted submission body.
That should be impossible.

Much as I would like to, I have no way of investigating this error
without compromising your data set and private encryption key.

The work-around is to discard this problematic submission from your
result set by navigating into the ODK Briefcase Storage directory (this is
generally located in your user directory. E.g., for Windows, if you open a
file browser, click on Downloads, then navigate up one directory, you will
be in your user directory and should see an ODK Briefcase Storage directory
there.

Within that directory, you would navigate into
forms/YOUR_FORM_ID/instances/

Simply move the uuid35235650-cd0b-43a0-b4ff-0d98dca480ff directory out of
the ODK Briefcase Storage directory.

You should then be able to decrypt and export the remaining surveys.

On Wed, Mar 11, 2015 at 6:27 PM, jennifer.linnea@gmail.com wrote:

Thanks for the quick response. I am already using those versions and
have never used the Mark-as-Complete action on the server. In fact, I have
never noticed that action.

The error seems to be associated with one instance, but there are a
number that are on the server and not decrypted after this failure. Any
other thoughts?

Starting Export...
Processing instance: uuid071ebb87-435a-44b7-b251-f528e58951b0
Processing instance: uuid101cfa87-7abd-473d-b56a-6baaa251d73d
Processing instance: uuid1b54a30b-b221-4134-8737-f77e6dcd2e2f
Processing instance: uuid2289894b-5aaf-45ca-8c55-98b4fe4e923c
Processing instance: uuid23524cb1-2ca2-4cf0-8333-a53499f18403
Processing instance: uuid35235650-cd0b-43a0-b4ff-0d98dca480ff
Error decrypting submission uuid35235650-cd0b-43a0-b4ff-0d98dca480ff
Cause: org.opendatakit.briefcase.model.FileSystemException: Error
decrypting:.temp Cause: form instance metadata differs from that in manifest
FAILED!

On Wednesday, 11 March 2015 18:07:09 UTC-7, Mitch Sundt wrote:

That issue was resolved in the 1.4.5 ODK Aggregate and 1.4.5 ODK
Briefcase releases (you must upgrade to both).

It relates to using the Mark-as-Complete action on the server when
using encrypted forms.

Mitch

On Wed, Mar 11, 2015 at 5:05 PM, jennife...@gmail.com wrote:
Hi

I am getting this same error now decrypting a form. Was there ever any
resolution on this issue?

Cause: org.opendatakit.briefcase.model.FileSystemException: Error
decrypting:.temp Cause: form instance metadata differs from that in manifest

Cheers,

Jenny

On Thursday, 6 November 2014 03:10:16 UTC-8, Shreya Agrawal wrote:

Hey Mitch,

Now we get a new error:

Error decrypting submission uuid9f4f0fe6-b2b4-43ce-91a5-1ffb67a63086
Cause: org.opendatakit.briefcase.model.FileSystemException: Error
decrypting:.temp Cause: form instance metadata differs from that in manifest

FAILED!

And we can't get access to alot of the data that we have already
downloaded from the Briefcase before.

We are successfully able to Pull ALL the data, but its the exporting
process that is causing the problems. Can we please discuss when you would
be able to work on the ODK Briefcase issue for us? We need to figure out
what's the best way for us to handle the situation on the data collection
end. Should we continue to do data collection or should we stop? We are
afraid that if we continue, we might not be able to get that data.

Thanks so much for your help!

Shreya

On Wednesday, November 5, 2014 1:49:31 AM UTC-5, Shreya Agrawal wrote:

Thanks Mitch! Should we connect via email on this?

Also, if you fix the Briefcase, will that fix this problem and we
can have access to rest of our data? If you think it will be a reoccurring
problem, we can create a new survey for data collectors to start using so
that the rest of the data does not get corrupted. I am just trying to
understand the severity of the problem.

Thanks!

On Tuesday, November 4, 2014 6:39:16 PM UTC-5, Mitch Sundt wrote:

I will be able to work on this next week, and can send you a new ODK
Briefcase without this issue (it might omit the problematic survey from the
csv, but will not cause the csv output to stop).

On Tue, Nov 4, 2014 at 8:14 AM, Shreya Agrawal s.agra...@gmail.com wrote:

Hey Mitch,

Thanks a lot. This is much more clear now.

I went back and double checked a few things now:

  1. ODK Aggregate/Form Management/Submission Admin tab only has 2
    surveys that are there. I will not mark them as complete and leave them as
    is.
  1. Briefcase is downloading data for those files that don't have an
    image. So we have 1,743 submissions and only 1,600 images. So that's not
    an issue
  1. Briefcase does continue to download other data (the new one), but
    as soon as we hit the error for the particular image, it stops (its the
    same unique number ID that keeps coming up). So we have about 151 surveys
    after that particular image that we cannot download. I can see all the
    files that have been submitted on ODK Aggregate, but they are all
    encrypted. So I have no idea which image ODK Briefcase is referring to and
    how I can manually delete it from the Briefcase so that I can still access
    the 151 surveys.

Any thoughts?

Thanks

On Monday, November 3, 2014 2:33:18 PM UTC-5, Mitch Sundt wrote:

No. Not having captured a signature would not be the reason for the
error. But if you did capture an image, but it was not preserved through
the submission process, then that would be a problem.

The "Mark as Complete" button is on the ODK Aggregate server, under
the Forms Management / Submissions Admin tab; it IS NOT the "Finalize"
button on ODK Collect.

My comment refers specifically to any submissions displayed under
the Forms Management / Submissions Admin tab. If you have submissions
displaying on that tab, then they are missing media attachments (e.g.,
signature files that were captured, but failed to be sent to the server) --
the sending of the submission to the server somehow failed. The user on the
device will need to re-send those submissions and their attachments.
Choosing "Mark as Complete" will cause problems during the decryption of
the affected submission.

If your briefcase storage location does have submissions that are
missing media attachments (and it sounds like it does), I believe it will
affect just those submissions, and not the pulling of data or the
generation of the full csv file of all results -- but I have not tested
this. Please confirm that this is true through your own testing. I do not
know if the affected data will be emitted into the csv or not. I think it
is likely that the affected submissions will not have their data written to
the csv, but I have not confirmed this.

On Sat, Nov 1, 2014 at 12:03 PM, Shreya Agrawal s.agra...@gmail.com wrote:

Hey Mitch,

Quick clarification: the signature files don't exist for these
surveys because we never captured them. If the person does not give
consent, we don't have a signature so there are no image files. Therefore,
when we are downloading the data, all these files don't have an image. Is
that the reason for the error?

If we don't Mark as complete - how do the data collectors upload the
data? These are the files that are finalized?

I wanted to know how we can still download all these surveys that
don't have an image through the briefcase without the error. Is that
possible?

Also, we have the latest version of ODK Downloaded that's availble
on the Google store.

Can you please clarify?

Thanks

Shreya

--

--

Post: opend...@googlegroups.com

Unsubscribe: opendatakit...@googlegroups.com

Options: http://groups.google.com/group/opendatakit?hl=en

You received this message because you are subscribed to the Google
Groups "ODK Community" group.

To unsubscribe from this group and stop receiving emails from it,
send an email to opendatakit...@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--

Mitch Sundt

Software Engineer

University of Washington

mitche...@gmail.com

--

--

Post: opend...@googlegroups.com

Unsubscribe: opendatakit...@googlegroups.com

Options: http://groups.google.com/group/opendatakit?hl=en

You received this message because you are subscribed to the Google
Groups "ODK Community" group.

To unsubscribe from this group and stop receiving emails from it,
send an email to opendatakit...@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--

Mitch Sundt

Software Engineer

University of Washington

mitche...@gmail.com

--

--

Post: opend...@googlegroups.com

Unsubscribe: opendatakit...@googlegroups.com

Options: http://groups.google.com/group/opendatakit?hl=en

You received this message because you are subscribed to the Google
Groups "ODK Community" group.

To unsubscribe from this group and stop receiving emails from it, send
an email to opendatakit...@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--

Mitch Sundt
Software Engineer
University of Washington
mitche...@gmail.com

--

Post: opendatakit@googlegroups.com
Unsubscribe: opendatakit+unsubscribe@googlegroups.com
Options: http://groups.google.com/group/opendatakit?hl=en

You received this message because you are subscribed to the Google
Groups "ODK Community" group.
To unsubscribe from this group and stop receiving emails from it, send
an email to opendatakit+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--
Mitch Sundt
Software Engineer
University of Washington
mitchellsundt@gmail.com

--

Post: opendatakit@googlegroups.com
Unsubscribe: opendatakit+unsubscribe@googlegroups.com
Options: http://groups.google.com/group/opendatakit?hl=en

You received this message because you are subscribed to a topic in the
Google Groups "ODK Community" group.
To unsubscribe from this topic, visit
https://groups.google.com/d/topic/opendatakit/IE7--7ys85o/unsubscribe.
To unsubscribe from this group and all its topics, send an email to
opendatakit+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--

Post: opendatakit@googlegroups.com
Unsubscribe: opendatakit+unsubscribe@googlegroups.com
Options: http://groups.google.com/group/opendatakit?hl=en

You received this message because you are subscribed to the Google Groups
"ODK Community" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to opendatakit+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--
Mitch Sundt
Software Engineer
University of Washington
mitchellsundt@gmail.com

15

Hi Mitch,

Just as an update, we have now had this happen again, on our (different)
encrypted form for another study.

I think at this point we need to give up the encryption.Is there a chance
that it could be a problem with one tablet or something?

Thanks,
Jenny

··· On Thu, Mar 12, 2015 at 11:14 AM, Mitch Sundt wrote:

On the plus side, this is the first time I have ever heard of anyone
getting this error. And a number of sites are actively using encrypted
surveys.

If you are willing, you can copy your entire ODK Briefcase Storage
directory, then delete all the forms except this problematic one, and,
within this problematic form, delete all instances except the one that was
causing problems.

Then zip up that slimmed-down directory and your private key and send it
directly to me ( mitchellsundt@gmail.com ).

It is unlikely that I will be able to resolve this issue anytime soon, as,
even with this information, I suspect it won't be enough to indicate a root
cause for the problem.

On Thu, Mar 12, 2015 at 10:55 AM, Jenny Smith jennifer.linnea@gmail.com wrote:

Thanks Mitch,

That has worked, so we only have that one record missing from the
database now.

It is now sounding like we may not even need the encryption, which we had
done as a precaution. It may be possible to share access with you if you
think there might be a way to recover that one record. I am just a little
nervous of keeping the encryption if this is a problem with the form or
tablets.

Cheers,
Jenny

On Thu, Mar 12, 2015 at 10:41 AM, Mitch Sundt mitchellsundt@gmail.com wrote:

Hmm.

The code:

boolean same =

decryptedSubmission.formId.equals(encryptionManifest.formId);

if ( !same ) {
  throw new FileSystemException("Error decrypting:" +

unencryptedDir.getName()
+ " Cause: form instance metadata differs from that in
manifest");
}

Which is only possible if there is something very bizarre going on --
this indicates that the formId identified in the encrypted-submission
manifest does not match the formId within the decrypted submission body.
That should be impossible.

Much as I would like to, I have no way of investigating this error
without compromising your data set and private encryption key.

The work-around is to discard this problematic submission from your
result set by navigating into the ODK Briefcase Storage directory (this is
generally located in your user directory. E.g., for Windows, if you open a
file browser, click on Downloads, then navigate up one directory, you will
be in your user directory and should see an ODK Briefcase Storage directory
there.

Within that directory, you would navigate into
forms/YOUR_FORM_ID/instances/

Simply move the uuid35235650-cd0b-43a0-b4ff-0d98dca480ff directory out
of the ODK Briefcase Storage directory.

You should then be able to decrypt and export the remaining surveys.

On Wed, Mar 11, 2015 at 6:27 PM, jennifer.linnea@gmail.com wrote:

Thanks for the quick response. I am already using those versions and
have never used the Mark-as-Complete action on the server. In fact, I have
never noticed that action.

The error seems to be associated with one instance, but there are a
number that are on the server and not decrypted after this failure. Any
other thoughts?

Starting Export...
Processing instance: uuid071ebb87-435a-44b7-b251-f528e58951b0
Processing instance: uuid101cfa87-7abd-473d-b56a-6baaa251d73d
Processing instance: uuid1b54a30b-b221-4134-8737-f77e6dcd2e2f
Processing instance: uuid2289894b-5aaf-45ca-8c55-98b4fe4e923c
Processing instance: uuid23524cb1-2ca2-4cf0-8333-a53499f18403
Processing instance: uuid35235650-cd0b-43a0-b4ff-0d98dca480ff
Error decrypting submission uuid35235650-cd0b-43a0-b4ff-0d98dca480ff
Cause: org.opendatakit.briefcase.model.FileSystemException: Error
decrypting:.temp Cause: form instance metadata differs from that in manifest
FAILED!

On Wednesday, 11 March 2015 18:07:09 UTC-7, Mitch Sundt wrote:

That issue was resolved in the 1.4.5 ODK Aggregate and 1.4.5 ODK
Briefcase releases (you must upgrade to both).

It relates to using the Mark-as-Complete action on the server when
using encrypted forms.

Mitch

On Wed, Mar 11, 2015 at 5:05 PM, jennife...@gmail.com wrote:
Hi

I am getting this same error now decrypting a form. Was there ever
any resolution on this issue?

Cause: org.opendatakit.briefcase.model.FileSystemException: Error
decrypting:.temp Cause: form instance metadata differs from that in manifest

Cheers,

Jenny

On Thursday, 6 November 2014 03:10:16 UTC-8, Shreya Agrawal wrote:

Hey Mitch,

Now we get a new error:

Error decrypting submission
uuid9f4f0fe6-b2b4-43ce-91a5-1ffb67a63086 Cause:
org.opendatakit.briefcase.model.FileSystemException: Error decrypting:.temp
Cause: form instance metadata differs from that in manifest

FAILED!

And we can't get access to alot of the data that we have already
downloaded from the Briefcase before.

We are successfully able to Pull ALL the data, but its the
exporting process that is causing the problems. Can we please discuss when
you would be able to work on the ODK Briefcase issue for us? We need to
figure out what's the best way for us to handle the situation on the data
collection end. Should we continue to do data collection or should we
stop? We are afraid that if we continue, we might not be able to get that
data.

Thanks so much for your help!

Shreya

On Wednesday, November 5, 2014 1:49:31 AM UTC-5, Shreya Agrawal wrote:

Thanks Mitch! Should we connect via email on this?

Also, if you fix the Briefcase, will that fix this problem and we
can have access to rest of our data? If you think it will be a reoccurring
problem, we can create a new survey for data collectors to start using so
that the rest of the data does not get corrupted. I am just trying to
understand the severity of the problem.

Thanks!

On Tuesday, November 4, 2014 6:39:16 PM UTC-5, Mitch Sundt wrote:

I will be able to work on this next week, and can send you a new
ODK Briefcase without this issue (it might omit the problematic survey from
the csv, but will not cause the csv output to stop).

On Tue, Nov 4, 2014 at 8:14 AM, Shreya Agrawal s.agra...@gmail.com wrote:

Hey Mitch,

Thanks a lot. This is much more clear now.

I went back and double checked a few things now:

  1. ODK Aggregate/Form Management/Submission Admin tab only has 2
    surveys that are there. I will not mark them as complete and leave them as
    is.
  1. Briefcase is downloading data for those files that don't have an
    image. So we have 1,743 submissions and only 1,600 images. So that's not
    an issue
  1. Briefcase does continue to download other data (the new one),
    but as soon as we hit the error for the particular image, it stops (its the
    same unique number ID that keeps coming up). So we have about 151 surveys
    after that particular image that we cannot download. I can see all the
    files that have been submitted on ODK Aggregate, but they are all
    encrypted. So I have no idea which image ODK Briefcase is referring to and
    how I can manually delete it from the Briefcase so that I can still access
    the 151 surveys.

Any thoughts?

Thanks

On Monday, November 3, 2014 2:33:18 PM UTC-5, Mitch Sundt wrote:

No. Not having captured a signature would not be the reason for the
error. But if you did capture an image, but it was not preserved through
the submission process, then that would be a problem.

The "Mark as Complete" button is on the ODK Aggregate server, under
the Forms Management / Submissions Admin tab; it IS NOT the "Finalize"
button on ODK Collect.

My comment refers specifically to any submissions displayed under
the Forms Management / Submissions Admin tab. If you have submissions
displaying on that tab, then they are missing media attachments (e.g.,
signature files that were captured, but failed to be sent to the server) --
the sending of the submission to the server somehow failed. The user on the
device will need to re-send those submissions and their attachments.
Choosing "Mark as Complete" will cause problems during the decryption of
the affected submission.

If your briefcase storage location does have submissions that are
missing media attachments (and it sounds like it does), I believe it will
affect just those submissions, and not the pulling of data or the
generation of the full csv file of all results -- but I have not tested
this. Please confirm that this is true through your own testing. I do not
know if the affected data will be emitted into the csv or not. I think it
is likely that the affected submissions will not have their data written to
the csv, but I have not confirmed this.

On Sat, Nov 1, 2014 at 12:03 PM, Shreya Agrawal < s.agra...@gmail.com> wrote:

Hey Mitch,

Quick clarification: the signature files don't exist for these
surveys because we never captured them. If the person does not give
consent, we don't have a signature so there are no image files. Therefore,
when we are downloading the data, all these files don't have an image. Is
that the reason for the error?

If we don't Mark as complete - how do the data collectors upload
the data? These are the files that are finalized?

I wanted to know how we can still download all these surveys that
don't have an image through the briefcase without the error. Is that
possible?

Also, we have the latest version of ODK Downloaded that's availble
on the Google store.

Can you please clarify?

Thanks

Shreya

--

--

Post: opend...@googlegroups.com

Unsubscribe: opendatakit...@googlegroups.com

Options: http://groups.google.com/group/opendatakit?hl=en

You received this message because you are subscribed to the Google
Groups "ODK Community" group.

To unsubscribe from this group and stop receiving emails from it,
send an email to opendatakit...@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--

Mitch Sundt

Software Engineer

University of Washington

mitche...@gmail.com

--

--

Post: opend...@googlegroups.com

Unsubscribe: opendatakit...@googlegroups.com

Options: http://groups.google.com/group/opendatakit?hl=en

You received this message because you are subscribed to the Google
Groups "ODK Community" group.

To unsubscribe from this group and stop receiving emails from it,
send an email to opendatakit...@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--

Mitch Sundt

Software Engineer

University of Washington

mitche...@gmail.com

--

--

Post: opend...@googlegroups.com

Unsubscribe: opendatakit...@googlegroups.com

Options: http://groups.google.com/group/opendatakit?hl=en

You received this message because you are subscribed to the Google
Groups "ODK Community" group.

To unsubscribe from this group and stop receiving emails from it,
send an email to opendatakit...@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--

Mitch Sundt
Software Engineer
University of Washington
mitche...@gmail.com

--

Post: opendatakit@googlegroups.com
Unsubscribe: opendatakit+unsubscribe@googlegroups.com
Options: http://groups.google.com/group/opendatakit?hl=en

You received this message because you are subscribed to the Google
Groups "ODK Community" group.
To unsubscribe from this group and stop receiving emails from it, send
an email to opendatakit+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--
Mitch Sundt
Software Engineer
University of Washington
mitchellsundt@gmail.com

--

Post: opendatakit@googlegroups.com
Unsubscribe: opendatakit+unsubscribe@googlegroups.com
Options: http://groups.google.com/group/opendatakit?hl=en

You received this message because you are subscribed to a topic in the
Google Groups "ODK Community" group.
To unsubscribe from this topic, visit
https://groups.google.com/d/topic/opendatakit/IE7--7ys85o/unsubscribe.
To unsubscribe from this group and all its topics, send an email to
opendatakit+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--

Post: opendatakit@googlegroups.com
Unsubscribe: opendatakit+unsubscribe@googlegroups.com
Options: http://groups.google.com/group/opendatakit?hl=en

You received this message because you are subscribed to the Google Groups
"ODK Community" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to opendatakit+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--
Mitch Sundt
Software Engineer
University of Washington
mitchellsundt@gmail.com

--

Post: opendatakit@googlegroups.com
Unsubscribe: opendatakit+unsubscribe@googlegroups.com
Options: http://groups.google.com/group/opendatakit?hl=en

You received this message because you are subscribed to a topic in the
Google Groups "ODK Community" group.
To unsubscribe from this topic, visit
https://groups.google.com/d/topic/opendatakit/IE7--7ys85o/unsubscribe.
To unsubscribe from this group and all its topics, send an email to
opendatakit+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

16

My suspicion is that it is a single device, probably running an older
version of ODK Collect.
But I am at a complete loss as to how this particular error can happen.

··· On Tue, Mar 24, 2015 at 11:57 AM, Jenny Smith wrote:

Hi Mitch,

Just as an update, we have now had this happen again, on our (different)
encrypted form for another study.

I think at this point we need to give up the encryption.Is there a chance
that it could be a problem with one tablet or something?

Thanks,
Jenny

On Thu, Mar 12, 2015 at 11:14 AM, Mitch Sundt mitchellsundt@gmail.com wrote:

On the plus side, this is the first time I have ever heard of anyone
getting this error. And a number of sites are actively using encrypted
surveys.

If you are willing, you can copy your entire ODK Briefcase Storage
directory, then delete all the forms except this problematic one, and,
within this problematic form, delete all instances except the one that was
causing problems.

Then zip up that slimmed-down directory and your private key and send it
directly to me ( mitchellsundt@gmail.com ).

It is unlikely that I will be able to resolve this issue anytime soon,
as, even with this information, I suspect it won't be enough to indicate a
root cause for the problem.

On Thu, Mar 12, 2015 at 10:55 AM, Jenny Smith jennifer.linnea@gmail.com wrote:

Thanks Mitch,

That has worked, so we only have that one record missing from the
database now.

It is now sounding like we may not even need the encryption, which we
had done as a precaution. It may be possible to share access with you if
you think there might be a way to recover that one record. I am just a
little nervous of keeping the encryption if this is a problem with the form
or tablets.

Cheers,
Jenny

On Thu, Mar 12, 2015 at 10:41 AM, Mitch Sundt mitchellsundt@gmail.com wrote:

Hmm.

The code:

boolean same =

decryptedSubmission.formId.equals(encryptionManifest.formId);

if ( !same ) {
  throw new FileSystemException("Error decrypting:" +

unencryptedDir.getName()
+ " Cause: form instance metadata differs from that in
manifest");
}

Which is only possible if there is something very bizarre going on --
this indicates that the formId identified in the encrypted-submission
manifest does not match the formId within the decrypted submission body.
That should be impossible.

Much as I would like to, I have no way of investigating this error
without compromising your data set and private encryption key.

The work-around is to discard this problematic submission from your
result set by navigating into the ODK Briefcase Storage directory (this is
generally located in your user directory. E.g., for Windows, if you open a
file browser, click on Downloads, then navigate up one directory, you will
be in your user directory and should see an ODK Briefcase Storage directory
there.

Within that directory, you would navigate into
forms/YOUR_FORM_ID/instances/

Simply move the uuid35235650-cd0b-43a0-b4ff-0d98dca480ff directory out
of the ODK Briefcase Storage directory.

You should then be able to decrypt and export the remaining surveys.

On Wed, Mar 11, 2015 at 6:27 PM, jennifer.linnea@gmail.com wrote:

Thanks for the quick response. I am already using those versions and
have never used the Mark-as-Complete action on the server. In fact, I have
never noticed that action.

The error seems to be associated with one instance, but there are a
number that are on the server and not decrypted after this failure. Any
other thoughts?

Starting Export...
Processing instance: uuid071ebb87-435a-44b7-b251-f528e58951b0
Processing instance: uuid101cfa87-7abd-473d-b56a-6baaa251d73d
Processing instance: uuid1b54a30b-b221-4134-8737-f77e6dcd2e2f
Processing instance: uuid2289894b-5aaf-45ca-8c55-98b4fe4e923c
Processing instance: uuid23524cb1-2ca2-4cf0-8333-a53499f18403
Processing instance: uuid35235650-cd0b-43a0-b4ff-0d98dca480ff
Error decrypting submission uuid35235650-cd0b-43a0-b4ff-0d98dca480ff
Cause: org.opendatakit.briefcase.model.FileSystemException: Error
decrypting:.temp Cause: form instance metadata differs from that in manifest
FAILED!

On Wednesday, 11 March 2015 18:07:09 UTC-7, Mitch Sundt wrote:

That issue was resolved in the 1.4.5 ODK Aggregate and 1.4.5 ODK
Briefcase releases (you must upgrade to both).

It relates to using the Mark-as-Complete action on the server when
using encrypted forms.

Mitch

On Wed, Mar 11, 2015 at 5:05 PM, jennife...@gmail.com wrote:
Hi

I am getting this same error now decrypting a form. Was there ever
any resolution on this issue?

Cause: org.opendatakit.briefcase.model.FileSystemException: Error
decrypting:.temp Cause: form instance metadata differs from that in manifest

Cheers,

Jenny

On Thursday, 6 November 2014 03:10:16 UTC-8, Shreya Agrawal wrote:

Hey Mitch,

Now we get a new error:

Error decrypting submission
uuid9f4f0fe6-b2b4-43ce-91a5-1ffb67a63086 Cause:
org.opendatakit.briefcase.model.FileSystemException: Error decrypting:.temp
Cause: form instance metadata differs from that in manifest

FAILED!

And we can't get access to alot of the data that we have already
downloaded from the Briefcase before.

We are successfully able to Pull ALL the data, but its the
exporting process that is causing the problems. Can we please discuss when
you would be able to work on the ODK Briefcase issue for us? We need to
figure out what's the best way for us to handle the situation on the data
collection end. Should we continue to do data collection or should we
stop? We are afraid that if we continue, we might not be able to get that
data.

Thanks so much for your help!

Shreya

On Wednesday, November 5, 2014 1:49:31 AM UTC-5, Shreya Agrawal wrote:

Thanks Mitch! Should we connect via email on this?

Also, if you fix the Briefcase, will that fix this problem and we
can have access to rest of our data? If you think it will be a reoccurring
problem, we can create a new survey for data collectors to start using so
that the rest of the data does not get corrupted. I am just trying to
understand the severity of the problem.

Thanks!

On Tuesday, November 4, 2014 6:39:16 PM UTC-5, Mitch Sundt wrote:

I will be able to work on this next week, and can send you a new
ODK Briefcase without this issue (it might omit the problematic survey from
the csv, but will not cause the csv output to stop).

On Tue, Nov 4, 2014 at 8:14 AM, Shreya Agrawal < s.agra...@gmail.com> wrote:

Hey Mitch,

Thanks a lot. This is much more clear now.

I went back and double checked a few things now:

  1. ODK Aggregate/Form Management/Submission Admin tab only has 2
    surveys that are there. I will not mark them as complete and leave them as
    is.
  1. Briefcase is downloading data for those files that don't have
    an image. So we have 1,743 submissions and only 1,600 images. So that's
    not an issue
  1. Briefcase does continue to download other data (the new one),
    but as soon as we hit the error for the particular image, it stops (its the
    same unique number ID that keeps coming up). So we have about 151 surveys
    after that particular image that we cannot download. I can see all the
    files that have been submitted on ODK Aggregate, but they are all
    encrypted. So I have no idea which image ODK Briefcase is referring to and
    how I can manually delete it from the Briefcase so that I can still access
    the 151 surveys.

Any thoughts?

Thanks

On Monday, November 3, 2014 2:33:18 PM UTC-5, Mitch Sundt wrote:

No. Not having captured a signature would not be the reason for
the error. But if you did capture an image, but it was not preserved
through the submission process, then that would be a problem.

The "Mark as Complete" button is on the ODK Aggregate server,
under the Forms Management / Submissions Admin tab; it IS NOT the
"Finalize" button on ODK Collect.

My comment refers specifically to any submissions displayed under
the Forms Management / Submissions Admin tab. If you have submissions
displaying on that tab, then they are missing media attachments (e.g.,
signature files that were captured, but failed to be sent to the server) --
the sending of the submission to the server somehow failed. The user on the
device will need to re-send those submissions and their attachments.
Choosing "Mark as Complete" will cause problems during the decryption of
the affected submission.

If your briefcase storage location does have submissions that are
missing media attachments (and it sounds like it does), I believe it will
affect just those submissions, and not the pulling of data or the
generation of the full csv file of all results -- but I have not tested
this. Please confirm that this is true through your own testing. I do not
know if the affected data will be emitted into the csv or not. I think it
is likely that the affected submissions will not have their data written to
the csv, but I have not confirmed this.

On Sat, Nov 1, 2014 at 12:03 PM, Shreya Agrawal < s.agra...@gmail.com> wrote:

Hey Mitch,

Quick clarification: the signature files don't exist for these
surveys because we never captured them. If the person does not give
consent, we don't have a signature so there are no image files. Therefore,
when we are downloading the data, all these files don't have an image. Is
that the reason for the error?

If we don't Mark as complete - how do the data collectors upload
the data? These are the files that are finalized?

I wanted to know how we can still download all these surveys that
don't have an image through the briefcase without the error. Is that
possible?

Also, we have the latest version of ODK Downloaded that's availble
on the Google store.

Can you please clarify?

Thanks

Shreya

--

--

Post: opend...@googlegroups.com

Unsubscribe: opendatakit...@googlegroups.com

Options: http://groups.google.com/group/opendatakit?hl=en

You received this message because you are subscribed to the Google
Groups "ODK Community" group.

To unsubscribe from this group and stop receiving emails from it,
send an email to opendatakit...@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--

Mitch Sundt

Software Engineer

University of Washington

mitche...@gmail.com

--

--

Post: opend...@googlegroups.com

Unsubscribe: opendatakit...@googlegroups.com

Options: http://groups.google.com/group/opendatakit?hl=en

You received this message because you are subscribed to the Google
Groups "ODK Community" group.

To unsubscribe from this group and stop receiving emails from it,
send an email to opendatakit...@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--

Mitch Sundt

Software Engineer

University of Washington

mitche...@gmail.com

--

--

Post: opend...@googlegroups.com

Unsubscribe: opendatakit...@googlegroups.com

Options: http://groups.google.com/group/opendatakit?hl=en

You received this message because you are subscribed to the Google
Groups "ODK Community" group.

To unsubscribe from this group and stop receiving emails from it,
send an email to opendatakit...@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--

Mitch Sundt
Software Engineer
University of Washington
mitche...@gmail.com

--

Post: opendatakit@googlegroups.com
Unsubscribe: opendatakit+unsubscribe@googlegroups.com
Options: http://groups.google.com/group/opendatakit?hl=en

You received this message because you are subscribed to the Google
Groups "ODK Community" group.
To unsubscribe from this group and stop receiving emails from it, send
an email to opendatakit+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--
Mitch Sundt
Software Engineer
University of Washington
mitchellsundt@gmail.com

--

Post: opendatakit@googlegroups.com
Unsubscribe: opendatakit+unsubscribe@googlegroups.com
Options: http://groups.google.com/group/opendatakit?hl=en

You received this message because you are subscribed to a topic in the
Google Groups "ODK Community" group.
To unsubscribe from this topic, visit
https://groups.google.com/d/topic/opendatakit/IE7--7ys85o/unsubscribe.
To unsubscribe from this group and all its topics, send an email to
opendatakit+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--

Post: opendatakit@googlegroups.com
Unsubscribe: opendatakit+unsubscribe@googlegroups.com
Options: http://groups.google.com/group/opendatakit?hl=en

You received this message because you are subscribed to the Google
Groups "ODK Community" group.
To unsubscribe from this group and stop receiving emails from it, send
an email to opendatakit+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--
Mitch Sundt
Software Engineer
University of Washington
mitchellsundt@gmail.com

--

Post: opendatakit@googlegroups.com
Unsubscribe: opendatakit+unsubscribe@googlegroups.com
Options: http://groups.google.com/group/opendatakit?hl=en

You received this message because you are subscribed to a topic in the
Google Groups "ODK Community" group.
To unsubscribe from this topic, visit
https://groups.google.com/d/topic/opendatakit/IE7--7ys85o/unsubscribe.
To unsubscribe from this group and all its topics, send an email to
opendatakit+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--

Post: opendatakit@googlegroups.com
Unsubscribe: opendatakit+unsubscribe@googlegroups.com
Options: http://groups.google.com/group/opendatakit?hl=en

You received this message because you are subscribed to the Google Groups
"ODK Community" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to opendatakit+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--
Mitch Sundt
Software Engineer
University of Washington
mitchellsundt@gmail.com