ODK upload encrypted submission questions

Mitch_S · December 2, 2015, 8:20pm

(adding opendatakit-developers@, as this question should be asked and
followed-up on that list).

Encrypted forms are handled as ordinary form definitions up until the form
is marked as finalized.

At that point, the form's submission XML is encrypted into
submission.xml.enc and a new submission XML is constructed based upon a
form definition that describes an encrypted form, its encrypted
attachments, an encrypted representation of the single-use encryption key
that was used for that encryption, and the cryptographic signature of the
original form submission XML and its attachments.

The form definition that corresponds to the resulting submission XML is
defined in the source code of the XML form definition parsing logic here:

github.com

getodk/aggregate/blob/master/src/main/java/org/opendatakit/aggregate/parser/BaseFormParserForJavaRosa.java#L159


      
          // extracted from XForm during parsing
          private final Map<String, Integer> stringLengths = new HashMap<String, Integer>();
          // original bindings from parse-time for later comparison
          private final List<Element> bindElements = new ArrayList<Element>();
          
          /**
           * Alternate constructor for internally comparing whether two form definitions
           * share the same data elements and storage models. This just parses the
           * supplied xml and does nothing else.
           */
          protected BaseFormParserForJavaRosa(String existingXml, String existingTitle, boolean allowLegacy)
              throws ODKIncompleteSubmissionData {
            if (existingXml == null) {
              throw new ODKIncompleteSubmissionData(Reason.MISSING_XML);
            }
          
            xml = existingXml;
          
            initializeJavaRosa();
          
            Document doc = parseXmlToDocument(xml);

Note that this is not a fully-usable interactive form definition (you can't
give it to ODK Collect have have it work). It contains enough information
for ODK Aggregate to construct the storage structures for the encrypted
form. And it treats all the attachments as 'image/*' in the form
definition, though, when they are uploaded, they are all uploaded as
generic binary octet attachments (since they are encrypted).

The decryption of the form and the verification of the signature is handled
by this routine:

github.com

getodk/briefcase/blob/master/src/org/opendatakit/briefcase/util/FileSystemUtils.java#L738


      
          /*
           * Copyright (C) 2011 University of Washington.
           *
           * Licensed under the Apache License, Version 2.0 (the "License"); you may not
           * use this file except in compliance with the License. You may obtain a copy of
           * the License at
           *
           * http://www.apache.org/licenses/LICENSE-2.0
           *
           * Unless required by applicable law or agreed to in writing, software
           * distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
           * WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
           * License for the specific language governing permissions and limitations under
           * the License.
           */
          
          package org.opendatakit.briefcase.util;
          
          import java.io.File;
          import java.io.FileInputStream;
          import java.io.FileNotFoundException;
          import java.io.IOException;
          import java.io.InputStream;
          import java.math.BigInteger;
          import java.nio.file.Path;
          import java.security.MessageDigest;
          import java.security.NoSuchAlgorithmException;
          import java.sql.SQLException;
          import java.util.ArrayList;
          import java.util.List;
          import java.util.Objects;
          import org.opendatakit.briefcase.model.FileSystemException;
          import org.opendatakit.briefcase.model.OdkCollectFormDefinition;
          import org.slf4j.Logger;
          import org.slf4j.LoggerFactory;
          
          public class FileSystemUtils {
          
            private static final Logger log = LoggerFactory.getLogger(FileSystemUtils.class);
          
            public static final String FORMS_DIR = "forms";
            static final String INSTANCE_DIR = "instances";
            private static final String HSQLDB_DIR = "info.hsqldb";
            private static final String HSQLDB_DB = "info";
            private static final String SMALLSQL_DIR = "info.db";
            static final String HSQLDB_JDBC_PREFIX = "jdbc:hsqldb:file:";
          
            static final String SMALLSQL_JDBC_PREFIX = "jdbc:smallsql:";
          
            // Predicates to determine whether the folder is an ODK Device
            // ODK folder or underneath that folder.
          
            private static boolean isODKDevice(File pathname) {
              File fo = new File(pathname, "odk");
              File foi = new File(fo, "instances");
              File fof = new File(fo, "forms");
              return fo.exists() && foi.exists() && fof.exists();
            }
          
            public static boolean isUnderODKFolder(File pathname) {
              File parent = (pathname == null ? null : pathname.getParentFile());
              File current = pathname;
              while (parent != null) {
                if (isODKDevice(parent) && current.getName().equals("odk"))
                  return true;
                current = parent;
                parent = parent.getParentFile();
              }
              return false;
            }
          
            public static List<OdkCollectFormDefinition> getODKFormList(File odk) {
              List<OdkCollectFormDefinition> formsList = new ArrayList<>();
              File forms = new File(odk, "forms");
              if (forms.exists()) {
                File[] formDirs = forms.listFiles();
                for (File f : Objects.requireNonNull(formDirs)) {
                  if (f.isFile() && f.getName().endsWith(".xml")) {
                    try {
                      formsList.add(new OdkCollectFormDefinition(f));
                    } catch (BadFormDefinition e) {
                      log.debug("bad form definition", e);
                    }
                  }
                }
              }
              return formsList;
            }
          
            private static File getFormsFolder(File briefcaseFolder) {
              return new File(briefcaseFolder, FORMS_DIR);
            }
          
            // TODO Ensure this is OK, as opposed to using stripIllegalChars()
            private static String asFilesystemSafeName(String formName) {
              return formName.replaceAll("[/\\\\:]", "").trim();
            }
          
            public static File getFormDirectory(String formName, File briefcaseFolder)
                throws FileSystemException {
              // clean up friendly form name...
              String rootName = asFilesystemSafeName(formName);
              File formPath = new File(getFormsFolder(briefcaseFolder), rootName);
              if (!formPath.exists() && !formPath.mkdirs()) {
                throw new FileSystemException("unable to create directory: " + formPath.getAbsolutePath());
              }
              return formPath;
            }
          
            static String getFormDatabaseUrl(File formDirectory) throws FileSystemException {
          
              File oldDbFile = new File(formDirectory, SMALLSQL_DIR);
              File dbDir = new File(formDirectory, HSQLDB_DIR);
              File dbFile = new File(dbDir, HSQLDB_DB);
          
          
              if (!dbDir.exists()) {
                log.info("Creating database directory {}", dbDir);
                if (!dbDir.mkdirs()) {
                  log.warn("failed to create database directory");
                } else if (oldDbFile.exists()) {
                  migrateDatabase(oldDbFile, dbFile);
                }
              }
          
              return getJdbcUrl(dbFile);
            }
          
            private static void migrateDatabase(File oldDbFile, File dbFile) throws FileSystemException {
              try {
                DatabaseUtils.migrateData(getJdbcUrl(oldDbFile), getJdbcUrl(dbFile));
              } catch (SQLException e) {
                throw new FileSystemException(String.format("failed to migrate database %s to %s", oldDbFile, dbFile), e);
              }
              if (!oldDbFile.renameTo(getBackupFile(oldDbFile))) {
                throw new FileSystemException("failed to backup database after migration");
              }
            }
          
            private static File getBackupFile(File file) {
              return new File(file.getParent(), file.getName() + ".bak");
            }
          
            private static String getJdbcUrl(File dbFile) throws FileSystemException {
              if (isHypersonicDatabase(dbFile)) {
                return HSQLDB_JDBC_PREFIX + dbFile.getAbsolutePath();
              } else if (isSmallSQLDatabase(dbFile)) {
                return SMALLSQL_JDBC_PREFIX + dbFile.getAbsolutePath() + (dbFile.exists() ? "" : "?create=true");
              } else {
                throw new FileSystemException("unknown database type for file " + dbFile);
              }
            }
          
            private static boolean isSmallSQLDatabase(File dbFile) {
              return SMALLSQL_DIR.equals(dbFile.getName());
            }
          
            private static boolean isHypersonicDatabase(File dbFile) {
              File parentFile = dbFile.getParentFile();
              return HSQLDB_DB.equals(dbFile.getName()) && parentFile != null && HSQLDB_DIR.equals(parentFile.getName());
            }
          
            public static File getFormDefinitionFile(File formDirectory) throws FileSystemException {
              return new File(formDirectory, formDirectory.getName() + ".xml");
            }
          
            // May or may not actually exist on disk depending on whether the form definition has attached media.
            public static File getMediaDirectory(File formDirectory) throws FileSystemException {
              return new File(formDirectory, formDirectory.getName() + "-media");
            }
          
            static File getFormInstancesDirectory(File formDirectory) throws FileSystemException {
              File instancesDir = new File(formDirectory, INSTANCE_DIR);
              if (!instancesDir.exists() && !instancesDir.mkdirs()) {
                throw new FileSystemException("unable to create directory: " + instancesDir.getAbsolutePath());
              }
              return instancesDir;
            }
          
            static boolean isFormRelativeInstancePath(String path) {
              return path.startsWith(INSTANCE_DIR);
            }
          
            static Path makeRelative(File parent, File child) {
              Path parentPath = parent.toPath();
              Path childPath = child.toPath();
              return parentPath.relativize(childPath);
            }
          
            static File getFormSubmissionDirectory(File formInstancesDir, String instanceID) {
              // construct the instance directory File...
              String instanceDirName = asFilesystemSafeName(instanceID);
              File instanceDir = new File(formInstancesDir, instanceDirName);
              if (!instanceDir.exists() || instanceDir.isDirectory()) {
                return instanceDir;
              }
              return null;
            }
          
            public static String getMd5Hash(File file) {
              try {
                // CTS (6/15/2010) : stream file through digest instead of handing
                // it the
                // byte[]
                MessageDigest md = MessageDigest.getInstance("MD5");
                int chunkSize = 256;
          
                byte[] chunk = new byte[chunkSize];
          
                // Get the size of the file
                long lLength = file.length();
          
                if (lLength > Integer.MAX_VALUE) {
                  log.error("File is too large");
                  return null;
                }
          
                int length = (int) lLength;
          
                InputStream is;
                is = new FileInputStream(file);
          
                int l;
                for (l = 0; l + chunkSize < length; l += chunkSize) {
                  is.read(chunk, 0, chunkSize);
                  md.update(chunk, 0, chunkSize);
                }
          
                int remaining = length - l;
                if (remaining > 0) {
                  is.read(chunk, 0, remaining);
                  md.update(chunk, 0, remaining);
                }
                byte[] messageDigest = md.digest();
          
                BigInteger number = new BigInteger(1, messageDigest);
                String md5 = number.toString(16);
                while (md5.length() < 32)
                  md5 = "0" + md5;
                is.close();
                return md5;
          
              } catch (NoSuchAlgorithmException e) {
                log.error("MD5 calculation failed", e);
                return null;
          
              } catch (FileNotFoundException e) {
                log.error("No File", e);
                return null;
              } catch (IOException e) {
                log.error("Problem reading from file", e);
                return null;
              }
          
            }
          
          }

This file has been truncated. show original

We intentionally do not provide any means to decrypt the form on the
server.
It must be done through the ODK Briefcase app.

···

On Tue, Dec 1, 2015 at 10:43 PM, Tom Snyder wrote:

Hi, Mitchell,
I am wondering if you would answer a few questions for me. I am building
a jersey web service to accept an encrypted Kobo form which I've been told
is based on ODK. My 1st question is : does ODK send the submission.XML.enc
file as part of the upload? Is there open source code that can show me how
to access both the XML payload and the submission.XML.enc encrypted data as
it is received by the server?
I appreciate any guidance and help you can provide.

Tom

Kind thanks,
Tom Snyder
Mobile: 203-733-8281

http://www.smartstainable.org

http://www.geMatrix.com

--
Mitch Sundt
Software Engineer
University of Washington
mitchellsundt@gmail.com

Tom_Snyder · December 29, 2015, 4:51pm

Thank you Mitch,

Is the submission.xml.enc sent on each post to the server along with the
XML file ha contains the key, the file name and the digest?

Kind thanks,
Tom

···

On Wed, Dec 2, 2015 at 12:20 PM, Mitch Sundt wrote:

(adding opendatakit-developers@, as this question should be asked and
followed-up on that list).

Encrypted forms are handled as ordinary form definitions up until the form
is marked as finalized.

At that point, the form's submission XML is encrypted into
submission.xml.enc and a new submission XML is constructed based upon a
form definition that describes an encrypted form, its encrypted
attachments, an encrypted representation of the single-use encryption key
that was used for that encryption, and the cryptographic signature of the
original form submission XML and its attachments.

The form definition that corresponds to the resulting submission XML is
defined in the source code of the XML form definition parsing logic here:

https://github.com/opendatakit/aggregate/blob/master/src/main/java/org/opendatakit/aggregate/parser/BaseFormParserForJavaRosa.java#L159

Note that this is not a fully-usable interactive form definition (you
can't give it to ODK Collect have have it work). It contains enough
information for ODK Aggregate to construct the storage structures for the
encrypted form. And it treats all the attachments as 'image/*' in the form
definition, though, when they are uploaded, they are all uploaded as
generic binary octet attachments (since they are encrypted).

The decryption of the form and the verification of the signature is
handled by this routine:

https://github.com/opendatakit/briefcase/blob/master/src/org/opendatakit/briefcase/util/FileSystemUtils.java#L738

We intentionally do not provide any means to decrypt the form on the
server.
It must be done through the ODK Briefcase app.

On Tue, Dec 1, 2015 at 10:43 PM, Tom Snyder tom.snyder@gematrix.com wrote:

Hi, Mitchell,
I am wondering if you would answer a few questions for me. I am building
a jersey web service to accept an encrypted Kobo form which I've been told
is based on ODK. My 1st question is : does ODK send the submission.XML.enc
file as part of the upload? Is there open source code that can show me how
to access both the XML payload and the submission.XML.enc encrypted data as
it is received by the server?
I appreciate any guidance and help you can provide.

Tom

Kind thanks,
Tom Snyder
Mobile: 203-733-8281

http://www.smartstainable.org

http://www.geMatrix.com

--
Mitch Sundt
Software Engineer
University of Washington
mitchellsundt@gmail.com

--

Kind thanks,
Tom Snyder
Mobile: 203-733-8281

https://twitter.com/intent/follow?original_referer=https%3A%2F%2Fabout.twitter.com%2Fresources%2Fbuttons&region=follow_link&screen_name=smartstainable&tw_p=followbutton&variant=2.0

http://www.smartstainable.com/

Mitch_S · December 29, 2015, 5:30pm

Yes.

···

On Tue, Dec 29, 2015 at 8:51 AM, Tom Snyder wrote:

Thank you Mitch,

Is the submission.xml.enc sent on each post to the server along with the
XML file ha contains the key, the file name and the digest?

Kind thanks,
Tom

On Wed, Dec 2, 2015 at 12:20 PM, Mitch Sundt mitchellsundt@gmail.com wrote:

(adding opendatakit-developers@, as this question should be asked and
followed-up on that list).

Encrypted forms are handled as ordinary form definitions up until the
form is marked as finalized.

At that point, the form's submission XML is encrypted into
submission.xml.enc and a new submission XML is constructed based upon a
form definition that describes an encrypted form, its encrypted
attachments, an encrypted representation of the single-use encryption key
that was used for that encryption, and the cryptographic signature of the
original form submission XML and its attachments.

The form definition that corresponds to the resulting submission XML is
defined in the source code of the XML form definition parsing logic here:

https://github.com/opendatakit/aggregate/blob/master/src/main/java/org/opendatakit/aggregate/parser/BaseFormParserForJavaRosa.java#L159

Note that this is not a fully-usable interactive form definition (you
can't give it to ODK Collect have have it work). It contains enough
information for ODK Aggregate to construct the storage structures for the
encrypted form. And it treats all the attachments as 'image/*' in the form
definition, though, when they are uploaded, they are all uploaded as
generic binary octet attachments (since they are encrypted).

The decryption of the form and the verification of the signature is
handled by this routine:

https://github.com/opendatakit/briefcase/blob/master/src/org/opendatakit/briefcase/util/FileSystemUtils.java#L738

We intentionally do not provide any means to decrypt the form on the
server.
It must be done through the ODK Briefcase app.

On Tue, Dec 1, 2015 at 10:43 PM, Tom Snyder tom.snyder@gematrix.com wrote:

Hi, Mitchell,
I am wondering if you would answer a few questions for me. I am
building a jersey web service to accept an encrypted Kobo form which I've
been told is based on ODK. My 1st question is : does ODK send the
submission.XML.enc file as part of the upload? Is there open source code
that can show me how to access both the XML payload and the
submission.XML.enc encrypted data as it is received by the server?
I appreciate any guidance and help you can provide.

Tom

Kind thanks,
Tom Snyder
Mobile: 203-733-8281

http://www.smartstainable.org

http://www.geMatrix.com

--
Mitch Sundt
Software Engineer
University of Washington
mitchellsundt@gmail.com

--

Kind thanks,
Tom Snyder
Mobile: 203-733-8281

https://twitter.com/intent/follow?original_referer=https%3A%2F%2Fabout.twitter.com%2Fresources%2Fbuttons&region=follow_link&screen_name=smartstainable&tw_p=followbutton&variant=2.0

http://www.smartstainable.com/

--
Mitch Sundt
Software Engineer
University of Washington
mitchellsundt@gmail.com

Tom_Snyder · January 2, 2016, 11:53pm

Hi, Mitch,
I'm still a bit puzzled at this as I get the following in the Request Input
Stream for a given simple form:
--RZovx2LrfsYpGOT6gUHpiJQmMz0OCX
Content-Disposition: form-data; name="xml_submission_file";
filename="simplesec_2016-01-02_13-48-22.xml"
Content-Type: text/xml
Content-Transfer-Encoding: binary

rR1Xc90YuOCLx7iVjZvmLRy+YSfoMcpRX+WfWrKFS1oz2RWYR5fTyn3sZoHsfy+ZHnug/rgRayAdx/RYuYDtzCmUdr93sGAzI4aefg2sfRE08uukDTSl7vCNQq1p0fDN3oZQZ26Ql2BwxjCxubU+fFzMssXqnwjRwsyQjZbK8ieJ4Gf/GYHY0478oi2olAU/fPQtxdF1j1TNgNb3MelbLCGcJ12cjkBYYKPQDk1daRao10WypUbJOtG0Mc+r4K4VtSdr7xEukzIRw9kxaqZZj18sO7/KbWaYMvKH8DURvegFiDnf8ZZomH7kanhhjcAplXyXChLAqwjnZL9jLWPfgg==<orx:meta
xmlns:orx="http://openrosa.org/xforms
">orx:instanceIDuuid:6f573414-34e3-4d8f-a9c4-3260651b16b9</orx:instanceID></orx:meta>
submission.xml.encY2dc/n3NSV01GtyZHA7rxITk/grfdNo0dsRBmm97zY0I8OWtIoftlXTFR2BRXpoQMt14VcQ21WEpRU/J5gkFs//VGAd/a4+pgimc6qk2iaOq91fF/vpsYvosRjH3RtMvU+HcyfoKHEpCeqZAsW7+FF2qtWnEILchN1IeFHk3Mhw1Zybx3z/4mn8Ww2vC3Nm6rPOz0QwOpD9Jg26Rr9MRJz0pQAwe+V2RWsZPXM1ukdPDNy7RoPtJ5qQDacGxowGds86ZXJ8cS8djquTSCk7sdcH8sU+iK6CH8nX30CMzU6xdo8HGjnvEhj4iH7vHDjRymsQbGQLNz1VNHtq0WBIpWA==
--RZovx2LrfsYpGOT6gUHpiJQmMz0OCX--

I've taken apart the form and only find one part - the

But nowhere do I see the submission.xml.enc file included in the input
stream. Can you provide any guidance please.

Kind thanks,
Tom

···

On Tue, Dec 29, 2015 at 9:30 AM, Mitch Sundt wrote:

Yes.

On Tue, Dec 29, 2015 at 8:51 AM, Tom Snyder tom.snyder@gematrix.com wrote:

Thank you Mitch,

Is the submission.xml.enc sent on each post to the server along with the
XML file ha contains the key, the file name and the digest?

Kind thanks,
Tom

On Wed, Dec 2, 2015 at 12:20 PM, Mitch Sundt mitchellsundt@gmail.com wrote:

(adding opendatakit-developers@, as this question should be asked and
followed-up on that list).

Encrypted forms are handled as ordinary form definitions up until the
form is marked as finalized.

At that point, the form's submission XML is encrypted into
submission.xml.enc and a new submission XML is constructed based upon a
form definition that describes an encrypted form, its encrypted
attachments, an encrypted representation of the single-use encryption key
that was used for that encryption, and the cryptographic signature of the
original form submission XML and its attachments.

The form definition that corresponds to the resulting submission XML is
defined in the source code of the XML form definition parsing logic here:

https://github.com/opendatakit/aggregate/blob/master/src/main/java/org/opendatakit/aggregate/parser/BaseFormParserForJavaRosa.java#L159

Note that this is not a fully-usable interactive form definition (you
can't give it to ODK Collect have have it work). It contains enough
information for ODK Aggregate to construct the storage structures for the
encrypted form. And it treats all the attachments as 'image/*' in the form
definition, though, when they are uploaded, they are all uploaded as
generic binary octet attachments (since they are encrypted).

The decryption of the form and the verification of the signature is
handled by this routine:

https://github.com/opendatakit/briefcase/blob/master/src/org/opendatakit/briefcase/util/FileSystemUtils.java#L738

We intentionally do not provide any means to decrypt the form on the
server.
It must be done through the ODK Briefcase app.

On Tue, Dec 1, 2015 at 10:43 PM, Tom Snyder tom.snyder@gematrix.com wrote:

Hi, Mitchell,
I am wondering if you would answer a few questions for me. I am
building a jersey web service to accept an encrypted Kobo form which I've
been told is based on ODK. My 1st question is : does ODK send the
submission.XML.enc file as part of the upload? Is there open source code
that can show me how to access both the XML payload and the
submission.XML.enc encrypted data as it is received by the server?
I appreciate any guidance and help you can provide.

Tom

Kind thanks,
Tom Snyder
Mobile: 203-733-8281

http://www.smartstainable.org

http://www.geMatrix.com

--
Mitch Sundt
Software Engineer
University of Washington
mitchellsundt@gmail.com

--

Kind thanks,
Tom Snyder
Mobile: 203-733-8281

https://twitter.com/intent/follow?original_referer=https%3A%2F%2Fabout.twitter.com%2Fresources%2Fbuttons&region=follow_link&screen_name=smartstainable&tw_p=followbutton&variant=2.0

http://www.smartstainable.com/

--
Mitch Sundt
Software Engineer
University of Washington
mitchellsundt@gmail.com

--

Kind thanks,
Tom Snyder
Mobile: 203-733-8281

https://twitter.com/intent/follow?original_referer=https%3A%2F%2Fabout.twitter.com%2Fresources%2Fbuttons&region=follow_link&screen_name=smartstainable&tw_p=followbutton&variant=2.0

http://www.smartstainable.com/

Tom_Snyder · January 4, 2016, 12:17am

Please disregard the last request. The server was not recognizing multi
part.
Thank you,
Tom

Kind thanks,
Tom Snyder
Mobile: 203-733-8281

http://www.smartstainable.org

···

________________________________

On Jan 2, 2016 1:53 PM, "Tom Snyder" tom.snyder@gematrix.com wrote:

Hi, Mitch,
I'm still a bit puzzled at this as I get the following in the Request
Input Stream for a given simple form:
--RZovx2LrfsYpGOT6gUHpiJQmMz0OCX
Content-Disposition: form-data; name="xml_submission_file";
filename="simplesec_2016-01-02_13-48-22.xml"
Content-Type: text/xml
Content-Transfer-Encoding: binary

rR1Xc90YuOCLx7iVjZvmLRy+YSfoMcpRX+WfWrKFS1oz2RWYR5fTyn3sZoHsfy+ZHnug/rgRayAdx/RYuYDtzCmUdr93sGAzI4aefg2sfRE08uukDTSl7vCNQq1p0fDN3oZQZ26Ql2BwxjCxubU+fFzMssXqnwjRwsyQjZbK8ieJ4Gf/GYHY0478oi2olAU/fPQtxdF1j1TNgNb3MelbLCGcJ12cjkBYYKPQDk1daRao10WypUbJOtG0Mc+r4K4VtSdr7xEukzIRw9kxaqZZj18sO7/KbWaYMvKH8DURvegFiDnf8ZZomH7kanhhjcAplXyXChLAqwjnZL9jLWPfgg==<orx:meta
xmlns:orx="http://openrosa.org/xforms
">orx:instanceIDuuid:6f573414-34e3-4d8f-a9c4-3260651b16b9</orx:instanceID></orx:meta>

submission.xml.encY2dc/n3NSV01GtyZHA7rxITk/grfdNo0dsRBmm97zY0I8OWtIoftlXTFR2BRXpoQMt14VcQ21WEpRU/J5gkFs//VGAd/a4+pgimc6qk2iaOq91fF/vpsYvosRjH3RtMvU+HcyfoKHEpCeqZAsW7+FF2qtWnEILchN1IeFHk3Mhw1Zybx3z/4mn8Ww2vC3Nm6rPOz0QwOpD9Jg26Rr9MRJz0pQAwe+V2RWsZPXM1ukdPDNy7RoPtJ5qQDacGxowGds86ZXJ8cS8djquTSCk7sdcH8sU+iK6CH8nX30CMzU6xdo8HGjnvEhj4iH7vHDjRymsQbGQLNz1VNHtq0WBIpWA==
--RZovx2LrfsYpGOT6gUHpiJQmMz0OCX--

I've taken apart the form and only find one part - the

But nowhere do I see the submission.xml.enc file included in the input
stream. Can you provide any guidance please.

Kind thanks,
Tom

On Tue, Dec 29, 2015 at 9:30 AM, Mitch Sundt mitchellsundt@gmail.com wrote:

Yes.

On Tue, Dec 29, 2015 at 8:51 AM, Tom Snyder tom.snyder@gematrix.com wrote:

Thank you Mitch,

Is the submission.xml.enc sent on each post to the server along with the
XML file ha contains the key, the file name and the digest?

Kind thanks,
Tom

On Wed, Dec 2, 2015 at 12:20 PM, Mitch Sundt mitchellsundt@gmail.com wrote:

(adding opendatakit-developers@, as this question should be asked and
followed-up on that list).

Encrypted forms are handled as ordinary form definitions up until the
form is marked as finalized.

At that point, the form's submission XML is encrypted into
submission.xml.enc and a new submission XML is constructed based upon a
form definition that describes an encrypted form, its encrypted
attachments, an encrypted representation of the single-use encryption key
that was used for that encryption, and the cryptographic signature of the
original form submission XML and its attachments.

The form definition that corresponds to the resulting submission XML is
defined in the source code of the XML form definition parsing logic here:

https://github.com/opendatakit/aggregate/blob/master/src/main/java/org/opendatakit/aggregate/parser/BaseFormParserForJavaRosa.java#L159

Note that this is not a fully-usable interactive form definition (you
can't give it to ODK Collect have have it work). It contains enough
information for ODK Aggregate to construct the storage structures for the
encrypted form. And it treats all the attachments as 'image/*' in the form
definition, though, when they are uploaded, they are all uploaded as
generic binary octet attachments (since they are encrypted).

The decryption of the form and the verification of the signature is
handled by this routine:

https://github.com/opendatakit/briefcase/blob/master/src/org/opendatakit/briefcase/util/FileSystemUtils.java#L738

We intentionally do not provide any means to decrypt the form on the
server.
It must be done through the ODK Briefcase app.

On Tue, Dec 1, 2015 at 10:43 PM, Tom Snyder tom.snyder@gematrix.com wrote:

Hi, Mitchell,
I am wondering if you would answer a few questions for me. I am
building a jersey web service to accept an encrypted Kobo form which I've
been told is based on ODK. My 1st question is : does ODK send the
submission.XML.enc file as part of the upload? Is there open source code
that can show me how to access both the XML payload and the
submission.XML.enc encrypted data as it is received by the server?
I appreciate any guidance and help you can provide.

Tom

Kind thanks,
Tom Snyder
Mobile: 203-733-8281

http://www.smartstainable.org

http://www.geMatrix.com

--
Mitch Sundt
Software Engineer
University of Washington
mitchellsundt@gmail.com

--

Kind thanks,
Tom Snyder
Mobile: 203-733-8281

https://twitter.com/intent/follow?original_referer=https%3A%2F%2Fabout.twitter.com%2Fresources%2Fbuttons&region=follow_link&screen_name=smartstainable&tw_p=followbutton&variant=2.0

http://www.smartstainable.com/

--
Mitch Sundt
Software Engineer
University of Washington
mitchellsundt@gmail.com

--

Kind thanks,
Tom Snyder
Mobile: 203-733-8281

https://twitter.com/intent/follow?original_referer=https%3A%2F%2Fabout.twitter.com%2Fresources%2Fbuttons&region=follow_link&screen_name=smartstainable&tw_p=followbutton&variant=2.0

http://www.smartstainable.com/

Tom_Snyder · January 4, 2016, 5:14am

Please ignore the last request.
Thank you,
Tom

Kind thanks,
Tom Snyder
Mobile: 203-733-8281

http://www.smartstainable.org

···

________________________________

On Jan 2, 2016 1:53 PM, "Tom Snyder" tom.snyder@gematrix.com wrote:

Hi, Mitch,
I'm still a bit puzzled at this as I get the following in the Request Input
Stream for a given simple form:
--RZovx2LrfsYpGOT6gUHpiJQmMz0OCX
Content-Disposition: form-data; name="xml_submission_file";
filename="simplesec_2016-01-02_13-48-22.xml"
Content-Type: text/xml
Content-Transfer-Encoding: binary

rR1Xc90YuOCLx7iVjZvmLRy+YSfoMcpRX+WfWrKFS1oz2RWYR5fTyn3sZoHsfy+ZHnug/rgRayAdx/RYuYDtzCmUdr93sGAzI4aefg2sfRE08uukDTSl7vCNQq1p0fDN3oZQZ26Ql2BwxjCxubU+fFzMssXqnwjRwsyQjZbK8ieJ4Gf/GYHY0478oi2olAU/fPQtxdF1j1TNgNb3MelbLCGcJ12cjkBYYKPQDk1daRao10WypUbJOtG0Mc+r4K4VtSdr7xEukzIRw9kxaqZZj18sO7/KbWaYMvKH8DURvegFiDnf8ZZomH7kanhhjcAplXyXChLAqwjnZL9jLWPfgg==<orx:meta
xmlns:orx="http://openrosa.org/xforms
">orx:instanceIDuuid:6f573414-34e3-4d8f-a9c4-3260651b16b9</orx:instanceID></orx:meta>
submission.xml.encY2dc/n3NSV01GtyZHA7rxITk/grfdNo0dsRBmm97zY0I8OWtIoftlXTFR2BRXpoQMt14VcQ21WEpRU/J5gkFs//VGAd/a4+pgimc6qk2iaOq91fF/vpsYvosRjH3RtMvU+HcyfoKHEpCeqZAsW7+FF2qtWnEILchN1IeFHk3Mhw1Zybx3z/4mn8Ww2vC3Nm6rPOz0QwOpD9Jg26Rr9MRJz0pQAwe+V2RWsZPXM1ukdPDNy7RoPtJ5qQDacGxowGds86ZXJ8cS8djquTSCk7sdcH8sU+iK6CH8nX30CMzU6xdo8HGjnvEhj4iH7vHDjRymsQbGQLNz1VNHtq0WBIpWA==
--RZovx2LrfsYpGOT6gUHpiJQmMz0OCX--

I've taken apart the form and only find one part - the

But nowhere do I see the submission.xml.enc file included in the input
stream. Can you provide any guidance please.

Kind thanks,
Tom

On Tue, Dec 29, 2015 at 9:30 AM, Mitch Sundt mitchellsundt@gmail.com wrote:

Yes.

On Tue, Dec 29, 2015 at 8:51 AM, Tom Snyder tom.snyder@gematrix.com wrote:

Thank you Mitch,

Is the submission.xml.enc sent on each post to the server along with the
XML file ha contains the key, the file name and the digest?

Kind thanks,
Tom

On Wed, Dec 2, 2015 at 12:20 PM, Mitch Sundt mitchellsundt@gmail.com wrote:

(adding opendatakit-developers@, as this question should be asked and
followed-up on that list).

Encrypted forms are handled as ordinary form definitions up until the
form is marked as finalized.

At that point, the form's submission XML is encrypted into
submission.xml.enc and a new submission XML is constructed based upon a
form definition that describes an encrypted form, its encrypted
attachments, an encrypted representation of the single-use encryption key
that was used for that encryption, and the cryptographic signature of the
original form submission XML and its attachments.

The form definition that corresponds to the resulting submission XML is
defined in the source code of the XML form definition parsing logic here:

https://github.com/opendatakit/aggregate/blob/master/src/main/java/org/opendatakit/aggregate/parser/BaseFormParserForJavaRosa.java#L159

Note that this is not a fully-usable interactive form definition (you
can't give it to ODK Collect have have it work). It contains enough
information for ODK Aggregate to construct the storage structures for the
encrypted form. And it treats all the attachments as 'image/*' in the form
definition, though, when they are uploaded, they are all uploaded as
generic binary octet attachments (since they are encrypted).

The decryption of the form and the verification of the signature is
handled by this routine:

https://github.com/opendatakit/briefcase/blob/master/src/org/opendatakit/briefcase/util/FileSystemUtils.java#L738

We intentionally do not provide any means to decrypt the form on the
server.
It must be done through the ODK Briefcase app.

On Tue, Dec 1, 2015 at 10:43 PM, Tom Snyder tom.snyder@gematrix.com wrote:

Hi, Mitchell,
I am wondering if you would answer a few questions for me. I am
building a jersey web service to accept an encrypted Kobo form which I've
been told is based on ODK. My 1st question is : does ODK send the
submission.XML.enc file as part of the upload? Is there open source code
that can show me how to access both the XML payload and the
submission.XML.enc encrypted data as it is received by the server?
I appreciate any guidance and help you can provide.

Tom

Kind thanks,
Tom Snyder
Mobile: 203-733-8281

http://www.smartstainable.org

http://www.geMatrix.com

--
Mitch Sundt
Software Engineer
University of Washington
mitchellsundt@gmail.com

--

Kind thanks,
Tom Snyder
Mobile: 203-733-8281

https://twitter.com/intent/follow?original_referer=https%3A%2F%2Fabout.twitter.com%2Fresources%2Fbuttons&region=follow_link&screen_name=smartstainable&tw_p=followbutton&variant=2.0

http://www.smartstainable.com/

--
Mitch Sundt
Software Engineer
University of Washington
mitchellsundt@gmail.com

--

Kind thanks,
Tom Snyder
Mobile: 203-733-8281

https://twitter.com/intent/follow?original_referer=https%3A%2F%2Fabout.twitter.com%2Fresources%2Fbuttons&region=follow_link&screen_name=smartstainable&tw_p=followbutton&variant=2.0

http://www.smartstainable.com/