ODK Briefcase 1.18 export failing with incomplete encrypted submissions

Support
1

1. What is the problem? Be very detailed.

Export generates a CSV file with no data. This has been happening with other versions of ODK Briefcase. I thought with the "On export, skip submissions that can't be parsed" fix in ODK Briefcase 1.18 would work.

2. What app or server are you using and on what device and operating system? Include version numbers.

Windows 10

3. What you have you tried to fix the problem?

I have to delete the instance folder with missing .enc files. The export process then works

4. What steps can we take to reproduce the problem?

For encrypted data delete the .enc files in the instance folder. I am not sure why after the pull process, the instance folder only contains the submission.xml file.

5. Anything else we should know or have? If you have a test form or screenshots or logs, attach below.

From the customized log file the following error is generated;

Error exporting a form: org.opendatakit.briefcase.model.ParsingException: Encrypted file not found (see the logs for more info)

All operations completed

2

It sounds like you have some submissions on your server that are not encrypted. What server and client are you using?

3

@LN,

The enumerators are currently using Collect 1.27. The Server side is Aggregate 2.0. (this error has occurred in multiple versions of Collect)

The form is designed to encrypt data on finalizing the form. Is it possible some forms are encrypted and other fail?

Paul

4

Do you have a prior issue report that we could also look at?

It sounds like you end up with some submissions that have a submission.xml file but no submission.enc, is that correct? In that case, does the submission.xml file have any contents? Is it an encrypted submission manifest (e.g. https://getodk.github.io/xforms-spec/encryption#submission-manifest), an unencrypted submission, an empty or malformed file...? On the server, do those submissions show up as incomplete submissions (https://docs.getodk.org/aggregate-forms/#managing-submissions-manually)?

5

@LN,

I am not sure I had reported this earlier.

Yes, there a submmission.xml but without submission.enc, the Submission.xml is an encrypted manifest (see an example below).

<data encrypted="yes" id="IHIF20190301" instanceID="uuid:1bfc4d37-70ad-45cd-ab7c-588081478942" isComplete="true" markedAsCompleteDate="2020-05-12T11:11:59.987+03:00" submissionDate="2020-05-12T11:11:59.987+03:00" version="20190301" xmlns="http://opendatakit.org/submissions">
	<base64EncryptedKey>d/BUFhMmOGRtcFCFrKWT7rzjiO5tNPHmugt6/1yzOPVpSquUNAXPNHN8ffrRsxwjm8OAL1XiuvzSfND1yxFP6kD9WEyB+iYlDAxgrOGGpjZ9OzhPGNRk9sgknGdLsIn4nOp4STkHvBk24zYXDR7VWQILCoRPXqLCRDe2uGD4nAo706/EnKs+aRcty85/6ZhPs7cd00yULYNmtyQewD1gdo7TJO6i/9nFnmmqxMPXszq/zCofW5YYv0d43gowxmAJHH5YBmOointCnqxTcmQOoTQUj//8w8QCFGk55ut0vbfGroFH5bVfTpEbJbFxWjwfCVU4yxq9uDN/6KXMrh3CGA==</base64EncryptedKey>
	<n0:meta xmlns:n0="http://openrosa.org/xforms">
		<n0:instanceID>uuid:1bfc4d37-70ad-45cd-ab7c-588081478942</n0:instanceID>
	</n0:meta>
	<encryptedXmlFile/>
	<base64EncryptedElementSignature>YoosrSuJv8RuxVlizgrUJBvl8Bkbgybz1iMepkvRVt3vokTcArGCIrumeKgAi36p1a3FDkziE7031gW7T+UmUdGCgOnLZckcMsDzC6xKyfZfwiQD6MCnCuahIHz+QlMxiga25YgtGe/YhX6OIBB0fy1ZMPEUjD6zfHNCbIjr1vtihRPCUwRC0NwzMBSYpEXRx96WVd9dfCPB0cuuwSsuw+NT1wTt1BL0TBO6JEZDspj+TnEI4jmJnwh2tKGXkO1uppE1Bci9w7FXPCS8e/X9jkdsLFtieixYaSHJ1TvqK3M9+TNM1iK7S6l2yTN0QJvXTBLmhK4QQ6yjDTpnzcBwDw==</base64EncryptedElementSignature>
</data>

There are no submissions showing up as "incomplete submissions"

I hope I have provided enough detail.

Paul

6

Thanks very much for the detailed reproduction steps.

This looks to me like the submission was sent by Collect but not received by Aggregate. It's likely not showing up as incomplete because that detection is not implemented for encrypted forms (I'm not totally sure).

Is connectivity intermittent or weak where you are sending submissions from? To get the data, you would need to find who submitted those records and ask them to re-submit.

I have filed https://github.com/getodk/briefcase/issues/895 to at least keep the export from failing.

7

Thanks @LN,

Yes, some of the field sites have poor internet. however, the forms are shown as sent on ODK Collect and there is nothing to re-submit.

Thanks for filing the export failing with encrypted forms.

Asante
Paul

8

It is possible with a bad connection for Collect to think it has submitted everything but for the server not to have received it. If submissions are not deleted on send, https://docs.getodk.org/collect-forms/#sending-previously-sent-forms describes how to re-submit instances that Collect believes have succeeded but were incomplete.

9

@LN,

Thanks for the help so far.

I tried the process of resubmitting the forms from the device. The resubmission were successful. However, the encrypted files are still missing (when one attempts to export data). The forms fail to export.

What would be the cause of these files failing to be uploaded? Or maybe they don't exist? If they were not created during form finalization what could be the cause?

Paul

10

Was this resolved Paul. I have the same problem and using ODK briefcase 1.18 as well. The difference is, my submissions have 1 unencrypted record. Why is refusing to export all the records?
image

11

@nicred,

We were not successful in getting back this data.

Paul