Problem exporting from encrypted forms with Briefcase

I am having problems using encryption and I'm hoping someone can help me.

First, I built the form and checked that it worked correctly using ODK
Collect, that I could send the completed forms to my Aggregate server, and
then download and export them correctly using ODK Briefcase. All of that
worked.

Then I made a new form that was a copy of the original, but had a Public
Key specified. I can collect the data and upload it to Aggregate, where it
appears to be valid and encrypted.

I can pull the encrypted data from Aggregate using ODK Briefcase.

But when I try to extract the forms to CSV files using ODK Briefcase, it
gives me this error:

Starting Export...
Processing instance: uuid97f707d5-ef2d-46f9-8a49-23dc587b6771
Error decrypting submission uuid97f707d5-ef2d-46f9-8a49-23dc587b6771 Cause:
org.opendatakit.briefcase.model.FileSystemException: Error decrypting:
submission.xml Cause: org.opendatakit.briefcase.model.ParsingException:
Failed during parsing of submission Xml:
org.xmlpull.v1.XmlPullParserException: expected: '>' actual: ''
(position:END_TAG @1:881 in java.io.InputStreamReader@5ebb55e1)
FAILED!

If I open submission.xml in an editor, it looks like a properly formed XML
file. Does anyone have an idea what might be wrong here?

I'm using ODK Briefcase 1.3.2 Production with JRE 1.7.0_25 (64 bit) under
Windows 7 (64 bit) and I have copied the JCE Unlimited Policy 7 jars to my
jre7/lib/security folder.

submission.xml (1.27 KB)

Hi Jonathan,

Sounds like a bug. It's been filed at
https://code.google.com/p/opendatakit/issues/detail?id=918. Star the
issue to get updates.

Yaw

··· On Sat, Sep 21, 2013 at 4:00 PM, Jonathan Gilligan wrote: > I am having problems using encryption and I'm hoping someone can help me. > > First, I built the form and checked that it worked correctly using ODK > Collect, that I could send the completed forms to my Aggregate server, and > then download and export them correctly using ODK Briefcase. All of that > worked. > > Then I made a new form that was a copy of the original, but had a Public Key > specified. I can collect the data and upload it to Aggregate, where it > appears to be valid and encrypted. > > I can pull the encrypted data from Aggregate using ODK Briefcase. > > But when I try to extract the forms to CSV files using ODK Briefcase, it > gives me this error: > > Starting Export... > Processing instance: uuid97f707d5-ef2d-46f9-8a49-23dc587b6771 > Error decrypting submission uuid97f707d5-ef2d-46f9-8a49-23dc587b6771 Cause: > org.opendatakit.briefcase.model.FileSystemException: Error decrypting: > submission.xml Cause: org.opendatakit.briefcase.model.ParsingException: > Failed during parsing of submission Xml: > org.xmlpull.v1.XmlPullParserException: expected: '>' actual: '�' > (position:END_TAG @1:881 in java.io.InputStreamReader@5ebb55e1) > FAILED! > > If I open submission.xml in an editor, it looks like a properly formed XML > file. Does anyone have an idea what might be wrong here? > > I'm using ODK Briefcase 1.3.2 Production with JRE 1.7.0_25 (64 bit) under > Windows 7 (64 bit) and I have copied the JCE Unlimited Policy 7 jars to my > jre7/lib/security folder. > > -- > -- > Post: opendatakit@googlegroups.com > Unsubscribe: opendatakit+unsubscribe@googlegroups.com > Options: http://groups.google.com/group/opendatakit?hl=en > > --- > You received this message because you are subscribed to the Google Groups > "ODK Community" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to opendatakit+unsubscribe@googlegroups.com. > For more options, visit https://groups.google.com/groups/opt_out.

This seems to be a problem with the public encryption key you specified in
the XLSForm not matching the private key PEM file you are using for
decryption.

Try this with your own newly-generated public/private key files, rather
than the example ones. It may be the example ones are mismatched.

The mismatch would not be detected until you try exporting to CSV within
Briefcase.

··· On Sat, Sep 21, 2013 at 5:12 PM, Yaw Anokwa wrote:

Hi Jonathan,

Sounds like a bug. It's been filed at
https://code.google.com/p/opendatakit/issues/detail?id=918. Star the
issue to get updates.

Yaw

On Sat, Sep 21, 2013 at 4:00 PM, Jonathan Gilligan jonathan.gilligan@gmail.com wrote:

I am having problems using encryption and I'm hoping someone can help me.

First, I built the form and checked that it worked correctly using ODK
Collect, that I could send the completed forms to my Aggregate server,
and
then download and export them correctly using ODK Briefcase. All of that
worked.

Then I made a new form that was a copy of the original, but had a Public
Key
specified. I can collect the data and upload it to Aggregate, where it
appears to be valid and encrypted.

I can pull the encrypted data from Aggregate using ODK Briefcase.

But when I try to extract the forms to CSV files using ODK Briefcase, it
gives me this error:

Starting Export...
Processing instance: uuid97f707d5-ef2d-46f9-8a49-23dc587b6771
Error decrypting submission uuid97f707d5-ef2d-46f9-8a49-23dc587b6771
Cause:
org.opendatakit.briefcase.model.FileSystemException: Error decrypting:
submission.xml Cause: org.opendatakit.briefcase.model.ParsingException:
Failed during parsing of submission Xml:
org.xmlpull.v1.XmlPullParserException: expected: '>' actual: '�'
(position:END_TAG @1:881 in java.io.InputStreamReader@5ebb55e1)
FAILED!

If I open submission.xml in an editor, it looks like a properly formed
XML
file. Does anyone have an idea what might be wrong here?

I'm using ODK Briefcase 1.3.2 Production with JRE 1.7.0_25 (64 bit) under
Windows 7 (64 bit) and I have copied the JCE Unlimited Policy 7 jars to
my
jre7/lib/security folder.

--

Post: opendatakit@googlegroups.com
Unsubscribe: opendatakit+unsubscribe@googlegroups.com
Options: http://groups.google.com/group/opendatakit?hl=en


You received this message because you are subscribed to the Google Groups
"ODK Community" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to opendatakit+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

--

Post: opendatakit@googlegroups.com
Unsubscribe: opendatakit+unsubscribe@googlegroups.com
Options: http://groups.google.com/group/opendatakit?hl=en


You received this message because you are subscribed to the Google Groups
"ODK Community" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to opendatakit+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

--
Mitch Sundt
Software Engineer
University of Washington
mitchellsundt@gmail.com

My original problem report was using a freshly generated 2048-bit key pair
using OpenSSL following the procedure described in the ODK documents. I
just checked with OpenSSL rsa -text -noout -in MyPrivateKey.pem and OpenSSL
rsa -text -noout -pubin -in MyPublicKey.pem and verified that they have the
same modulus and exponent.

To put the public key in the form, I took the lines between ---BEGIN PUBLIC
KEY--- and ---END PUBLIC KEY--- in MyPublicKey.pem, removed the line breaks
so it's just one line of text, and put it in the form XML file in a
tag.

··· On Monday, September 23, 2013 1:38:32 PM UTC-5, Mitch Sundt wrote: > > This seems to be a problem with the public encryption key you specified in > the XLSForm not matching the private key PEM file you are using for > decryption. > > Try this with your own newly-generated public/private key files, rather > than the example ones. It may be the example ones are mismatched. > > The mismatch would not be detected until you try exporting to CSV within > Briefcase. > > > > > > On Sat, Sep 21, 2013 at 5:12 PM, Yaw Anokwa <yan...@nafundi.com wrote: > >> Hi Jonathan, >> >> Sounds like a bug. It's been filed at >> https://code.google.com/p/opendatakit/issues/detail?id=918. Star the >> issue to get updates. >> >> Yaw >> >> On Sat, Sep 21, 2013 at 4:00 PM, Jonathan Gilligan <jonathan...@gmail.com > wrote: >> > I am having problems using encryption and I'm hoping someone can help >> me. >> > >> > First, I built the form and checked that it worked correctly using ODK >> > Collect, that I could send the completed forms to my Aggregate server, >> and >> > then download and export them correctly using ODK Briefcase. All of that >> > worked. >> > >> > Then I made a new form that was a copy of the original, but had a >> Public Key >> > specified. I can collect the data and upload it to Aggregate, where it >> > appears to be valid and encrypted. >> > >> > I can pull the encrypted data from Aggregate using ODK Briefcase. >> > >> > But when I try to extract the forms to CSV files using ODK Briefcase, it >> > gives me this error: >> > >> > Starting Export... >> > Processing instance: uuid97f707d5-ef2d-46f9-8a49-23dc587b6771 >> > Error decrypting submission uuid97f707d5-ef2d-46f9-8a49-23dc587b6771 >> Cause: >> > org.opendatakit.briefcase.model.FileSystemException: Error decrypting: >> > submission.xml Cause: org.opendatakit.briefcase.model.ParsingException: >> > Failed during parsing of submission Xml: >> > org.xmlpull.v1.XmlPullParserException: expected: '>' actual: '�' >> > (position:END_TAG @1:881 in java.io.InputStreamReader@5ebb55e1) >> > FAILED! >> > >> > If I open submission.xml in an editor, it looks like a properly formed >> XML >> > file. Does anyone have an idea what might be wrong here? >> > >> > I'm using ODK Briefcase 1.3.2 Production with JRE 1.7.0_25 (64 bit) >> under >> > Windows 7 (64 bit) and I have copied the JCE Unlimited Policy 7 jars to >> my >> > jre7/lib/security folder. >> > >> > -- >> > -- >> > Post: opend...@googlegroups.com >> > Unsubscribe: opendatakit...@googlegroups.com >> > Options: http://groups.google.com/group/opendatakit?hl=en >> > >> > --- >> > You received this message because you are subscribed to the Google >> Groups >> > "ODK Community" group. >> > To unsubscribe from this group and stop receiving emails from it, send >> an >> > email to opendatakit...@googlegroups.com . >> > For more options, visit https://groups.google.com/groups/opt_out. >> >> -- >> -- >> Post: opend...@googlegroups.com >> Unsubscribe: opendatakit...@googlegroups.com >> Options: http://groups.google.com/group/opendatakit?hl=en >> >> --- >> You received this message because you are subscribed to the Google Groups >> "ODK Community" group. >> To unsubscribe from this group and stop receiving emails from it, send an >> email to opendatakit...@googlegroups.com . >> For more options, visit https://groups.google.com/groups/opt_out. >> > > > > -- > Mitch Sundt > Software Engineer > University of Washington > mitche...@gmail.com >

That sounds right. Can you either attach or send directly to me (
mitchellsundt@gmail.com ) the following files:

(1) the form definition file (XML)
(2) your private key PEM file
(3) the submission.xml.enc file for the submission that is causing the
problem.

Of course, since you'll be sending me the private key file, you will want
to regenerate a new key for any future work.

Thanks,

Mitch

··· On Mon, Sep 23, 2013 at 1:15 PM, Jonathan Gilligan < jonathan.gilligan@gmail.com> wrote:

My original problem report was using a freshly generated 2048-bit key pair
using OpenSSL following the procedure described in the ODK documents. I
just checked with OpenSSL rsa -text -noout -in MyPrivateKey.pem and OpenSSL
rsa -text -noout -pubin -in MyPublicKey.pem and verified that they have the
same modulus and exponent.

To put the public key in the form, I took the lines between ---BEGIN
PUBLIC KEY--- and ---END PUBLIC KEY--- in MyPublicKey.pem, removed the line
breaks so it's just one line of text, and put it in the form XML file in a
tag.

On Monday, September 23, 2013 1:38:32 PM UTC-5, Mitch Sundt wrote:

This seems to be a problem with the public encryption key you specified
in the XLSForm not matching the private key PEM file you are using for
decryption.

Try this with your own newly-generated public/private key files, rather
than the example ones. It may be the example ones are mismatched.

The mismatch would not be detected until you try exporting to CSV within
Briefcase.

On Sat, Sep 21, 2013 at 5:12 PM, Yaw Anokwa yan...@nafundi.com wrote:

Hi Jonathan,

Sounds like a bug. It's been filed at
https://code.google.com/p/**opendatakit/issues/detail?id=**918https://code.google.com/p/opendatakit/issues/detail?id=918.
Star the
issue to get updates.

Yaw

On Sat, Sep 21, 2013 at 4:00 PM, Jonathan Gilligan jonathan...@gmail.com wrote:

I am having problems using encryption and I'm hoping someone can help
me.

First, I built the form and checked that it worked correctly using ODK
Collect, that I could send the completed forms to my Aggregate server,
and
then download and export them correctly using ODK Briefcase. All of
that
worked.

Then I made a new form that was a copy of the original, but had a
Public Key
specified. I can collect the data and upload it to Aggregate, where it
appears to be valid and encrypted.

I can pull the encrypted data from Aggregate using ODK Briefcase.

But when I try to extract the forms to CSV files using ODK Briefcase,
it
gives me this error:

Starting Export...
Processing instance: uuid97f707d5-ef2d-46f9-8a49-**23dc587b6771
Error decrypting submission uuid97f707d5-ef2d-46f9-8a49-**23dc587b6771
Cause:
org.opendatakit.briefcase.model.FileSystemException: Error
decrypting:
submission.xml Cause: org.opendatakit.briefcase.

model.ParsingException:
Failed during parsing of submission Xml:
org.xmlpull.v1.**XmlPullParserException: expected: '>' actual: '�'
(position:END_TAG @1:881 in java.io.InputStreamReader@**5ebb55e1)
FAILED!

If I open submission.xml in an editor, it looks like a properly formed
XML
file. Does anyone have an idea what might be wrong here?

I'm using ODK Briefcase 1.3.2 Production with JRE 1.7.0_25 (64 bit)
under
Windows 7 (64 bit) and I have copied the JCE Unlimited Policy 7 jars
to my
jre7/lib/security folder.

--

Post: opend...@googlegroups.com
Unsubscribe: opendatakit...@**googlegroups.com

Options: http://groups.google.com/**group/opendatakit?hl=enhttp://groups.google.com/group/opendatakit?hl=en


You received this message because you are subscribed to the Google
Groups
"ODK Community" group.
To unsubscribe from this group and stop receiving emails from it, send
an
email to opendatakit...@**googlegroups.com.

For more options, visit https://groups.google.com/**groups/opt_outhttps://groups.google.com/groups/opt_out
.

--

Post: opend...@googlegroups.com
Unsubscribe: opendatakit...@**googlegroups.com

Options: http://groups.google.com/**group/opendatakit?hl=enhttp://groups.google.com/group/opendatakit?hl=en


You received this message because you are subscribed to the Google
Groups "ODK Community" group.
To unsubscribe from this group and stop receiving emails from it, send
an email to opendatakit...@**googlegroups.com.

For more options, visit https://groups.google.com/**groups/opt_outhttps://groups.google.com/groups/opt_out
.

--
Mitch Sundt
Software Engineer
University of Washington
mitche...@gmail.com

--

Post: opendatakit@googlegroups.com
Unsubscribe: opendatakit+unsubscribe@googlegroups.com
Options: http://groups.google.com/group/opendatakit?hl=en


You received this message because you are subscribed to the Google Groups
"ODK Community" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to opendatakit+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

--
Mitch Sundt
Software Engineer
University of Washington
mitchellsundt@gmail.com

Mitch,

A SurveyCTO client reported this problem on Friday, and we have worked on
it almost around the clock since then. It's a big deal. Here are some
details:

  1. We believe the problem to be Android 4.3's shift from Bouncy Castle to
    OpenSSL.

  2. If you look at the encrypted content coming from new devices, you will
    see that the sizes are not divisible by 16. There is an issue with the
    padding at the end.

  3. When the XML content is decrypted by Briefcase, it loses up to 16 bytes
    from the original content (basically, what would have been in the last
    encrypted block).

  4. Because the content at the very end of the XML file is basically
    deterministic, we made it so that our client software can auto-correct
    these cases (automatically add back the missing characters at the end of
    the file). This allows clients to decrypt and export data as if nothing
    whatsoever was wrong, and avoids any cases of possible data loss.

  5. I updated https://code.google.com/p/opendatakit/issues/detail?id=918with
    a temporary fix that requests use of Bouncy Castle instead of OpenSSL.
    That appears to work on one Nexus with 4.3 that we've tested, but we're not
    sure how guaranteed Bouncy Castle support is. The true fix is to correct
    the encryption code to properly encrypt the last block whether the provider
    is BC or OpenSSL.

If anybody with an active field deployment upgrades to Android 4.3, we
believe that they will run into this problem -- and they can lose data if
Briefcase isn't patched to auto-correct the data (in response to parsing
errors, it should try auto-correcting and retrying the parsing). We can
share our code for that and help you to patch Briefcase for ODK users. For
SurveyCTO users, we are already testing a new release.

We can discuss a final solution in
https://code.google.com/p/opendatakit/issues/detail?id=918.

Thanks,

Chris

··· On Mon, Sep 23, 2013 at 5:45 PM, Mitch Sundt wrote:

That sounds right. Can you either attach or send directly to me (
mitchellsundt@gmail.com ) the following files:

(1) the form definition file (XML)
(2) your private key PEM file
(3) the submission.xml.enc file for the submission that is causing the
problem.

Of course, since you'll be sending me the private key file, you will want
to regenerate a new key for any future work.

Thanks,

Mitch

On Mon, Sep 23, 2013 at 1:15 PM, Jonathan Gilligan < jonathan.gilligan@gmail.com> wrote:

My original problem report was using a freshly generated 2048-bit key
pair using OpenSSL following the procedure described in the ODK documents.
I just checked with OpenSSL rsa -text -noout -in MyPrivateKey.pem and
OpenSSL rsa -text -noout -pubin -in MyPublicKey.pem and verified that they
have the same modulus and exponent.

To put the public key in the form, I took the lines between ---BEGIN
PUBLIC KEY--- and ---END PUBLIC KEY--- in MyPublicKey.pem, removed the line
breaks so it's just one line of text, and put it in the form XML file in a
tag.

On Monday, September 23, 2013 1:38:32 PM UTC-5, Mitch Sundt wrote:

This seems to be a problem with the public encryption key you specified
in the XLSForm not matching the private key PEM file you are using for
decryption.

Try this with your own newly-generated public/private key files, rather
than the example ones. It may be the example ones are mismatched.

The mismatch would not be detected until you try exporting to CSV within
Briefcase.

On Sat, Sep 21, 2013 at 5:12 PM, Yaw Anokwa yan...@nafundi.com wrote:

Hi Jonathan,

Sounds like a bug. It's been filed at
https://code.google.com/p/**opendatakit/issues/detail?id=**918https://code.google.com/p/opendatakit/issues/detail?id=918.
Star the
issue to get updates.

Yaw

On Sat, Sep 21, 2013 at 4:00 PM, Jonathan Gilligan jonathan...@gmail.com wrote:

I am having problems using encryption and I'm hoping someone can help
me.

First, I built the form and checked that it worked correctly using ODK
Collect, that I could send the completed forms to my Aggregate
server, and
then download and export them correctly using ODK Briefcase. All of
that
worked.

Then I made a new form that was a copy of the original, but had a
Public Key
specified. I can collect the data and upload it to Aggregate, where it
appears to be valid and encrypted.

I can pull the encrypted data from Aggregate using ODK Briefcase.

But when I try to extract the forms to CSV files using ODK Briefcase,
it
gives me this error:

Starting Export...
Processing instance: uuid97f707d5-ef2d-46f9-8a49-**23dc587b6771
Error decrypting submission uuid97f707d5-ef2d-46f9-8a49-**23dc587b6771
Cause:
org.opendatakit.briefcase.model.FileSystemException: Error
decrypting:
submission.xml Cause: org.opendatakit.briefcase.

model.ParsingException:
Failed during parsing of submission Xml:
org.xmlpull.v1.XmlPullParserException: expected: '>' actual: '�'
(position:END_TAG @1:881 in java.io.InputStreamReader@

5ebb55e1)
FAILED!

If I open submission.xml in an editor, it looks like a properly
formed XML
file. Does anyone have an idea what might be wrong here?

I'm using ODK Briefcase 1.3.2 Production with JRE 1.7.0_25 (64 bit)
under
Windows 7 (64 bit) and I have copied the JCE Unlimited Policy 7 jars
to my
jre7/lib/security folder.

--

Post: opend...@googlegroups.com
Unsubscribe: opendatakit...@**googlegroups.com

Options: http://groups.google.com/**group/opendatakit?hl=enhttp://groups.google.com/group/opendatakit?hl=en


You received this message because you are subscribed to the Google
Groups
"ODK Community" group.
To unsubscribe from this group and stop receiving emails from it,
send an
email to opendatakit...@**googlegroups.com.

For more options, visit https://groups.google.com/**groups/opt_outhttps://groups.google.com/groups/opt_out
.

--

Post: opend...@googlegroups.com
Unsubscribe: opendatakit...@**googlegroups.com

Options: http://groups.google.com/**group/opendatakit?hl=enhttp://groups.google.com/group/opendatakit?hl=en


You received this message because you are subscribed to the Google
Groups "ODK Community" group.
To unsubscribe from this group and stop receiving emails from it, send
an email to opendatakit...@**googlegroups.com.

For more options, visit https://groups.google.com/**groups/opt_outhttps://groups.google.com/groups/opt_out
.

--
Mitch Sundt
Software Engineer
University of Washington
mitche...@gmail.com

--

Post: opendatakit@googlegroups.com
Unsubscribe: opendatakit+unsubscribe@googlegroups.com
Options: http://groups.google.com/group/opendatakit?hl=en


You received this message because you are subscribed to the Google Groups
"ODK Community" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to opendatakit+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

--
Mitch Sundt
Software Engineer
University of Washington
mitchellsundt@gmail.com

--

Post: opendatakit@googlegroups.com
Unsubscribe: opendatakit+unsubscribe@googlegroups.com
Options: http://groups.google.com/group/opendatakit?hl=en


You received this message because you are subscribed to the Google Groups
"ODK Community" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to opendatakit+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.