ODK Central - change to encrypted form once already started data collection breaks submissions

1. What is the problem? Be very detailed.

User project on ODK Central.
They collected X submissions via Enketo.
Then they turned on encryption for the Project.
Now you dont seem to be able to submit new forms.

If you create a new form within the project that works fine.

2. What app or server are you using and on what device and operating system? Include version numbers.

versions:
647569c54f6bbf26ea356eca0d14f7e5d1a89c6b
cddb691e40e84aabff87b9d427e22a50282d6f99 client (v1.1.2)
a33bc6fb3c34fe38894b0e9d0bb404f81da325e6 server (v1.1.1)

3. What you have you tried to fix the problem?

As outlined above new forms work fine, just the form which was already live and had some submissions before encryption doesnt work.

4. What steps can we take to reproduce the problem?

Try the above!

Hi @dr_michaelmarks!

What happens when they send new submissions? Does Central not accept the submission, or does a different issue arise? Is an error message shown?

Also, is Enketo being accessed by a web user from within Central, or is it being accessed from a public access link?

I tried to reproduce the issue, but I wasn't able to. If you have additional details about the issue or how to reproduce it, that would be very helpful. The steps I took were:

  • Create a new project. Upload and publish a form.
  • Create a public access link for the form.
  • Send a submission using Enketo as a web user, then send another submission using the public access link.
  • Encrypt the project.
  • Send another pair of submissions.
  • All four submissions are visible under the Submissions tab, and the more recent pair of submissions is marked as encrypted. I'm also able to download a CSV file.

Hi Matt

Is an error message shown?

Get a 404 Submission Server Not Found

Debugging console:
Failed to load resource: the server responded with a status of 404 (Not Found) -/submission/hMPIt3OyX9IJF4rUvHSugf61CjNgXRV?st=9gv98zRs7f2VQEYusRUmbOfDA76TPHr6hYRq5a9XcJgF1fiJ1Z7pvR0Nys0BPmVR:1

Also, is Enketo being accessed by a web user from within Central, or is it being accessed from a public access link?

Public Access Link

Interestingly I setup a new project on the same Central instance and didnt encounter this problem - but 2 different users at LSHTM (across 2 different Central instances) have separately reported it to me and I confirmed it for their specific project.
I have one server I could probably give you access to take a look if that might help?

Hi Matt

Further testing. I have found the specific error which is:
If your form doesnt have a settings tab with a stated FORM_ID then when you turn on encryption then form submission breaks via Enketo webforms.

You can reproduce this by

  1. Make 2 copies of a form
  2. On one of them have a settings tab of the XLS with a form ID (form name is optional, removing it doesnt seem to cause this behaviour as long as an ID is present)
  3. On the second (otherwise identical form) dont have a settings tab (i.e just the choices & surveys tab) (OR have one but leave FORM ID blank)
  4. Upload
  5. It doesnt matter if you do some submissions and then encrypt or just encrypt straight away - once encrypted the form without a settings tab will no longer be able to be submitted

This is true if form uploaded before/after encryption - but essentially without a settings tab and a FORM_ID then encryption breaks submission.

Proposed fix - a) when you are turning on encryption people need to be warned about this & b) when uploading forms if the XLS is missing a FORM ID we should warn people

cc @chrissyhroberts

2 Likes

Thank you for the additional details, @dr_michaelmarks! This is very helpful. What you describe is definitely not expected behavior. Project encryption should work even if there is a form whose XLSForm did not specify a form ID, and it should be possible to use a public access link to submit to such a form after the project is encrypted.

For some reason, I'm still having trouble reproducing the issue. I took the following steps on a server that uses the latest version of v1.1:

  • Create a new project.
  • Upload an XLSForm without a settings sheet.
  • Publish the form.
  • Create a public access link for the form.
  • Encrypt the project using the passphrase foobar123!.
  • Try to use the public access link to submit.
  • Enketo indicates that submission was successful, and I am able to download the submitted data.

When you try to reproduce the issue using your steps, are you able to do so consistently? If so, could you share an XLSForm that reproduces the issue? I'm also wondering whether you encounter the issue when you follow the steps I tried above?

Note that even if an XLSForm does not specify a form ID, every form does have an ID. If the XLSForm does not have a settings tab, then ODK Central sets the form ID to the filename of the XLSForm. You can see the form ID in the forms table on the project overview page.

Actually, we just found an issue that I think could be related to the one you're encountering: https://github.com/getodk/central-backend/issues/364. It seems like there are cases where if a form ID contains a space, then submitting via Enketo won't work after project encryption.

You mentioned uploading an XLSForm without a settings tab, which will use the filename of the XLSForm for the form ID. Did the filename contain a space?

Think we are getting there

If project is encrypted

  1. If no form ID but the form name has no spaces then the system makes a form_ID based on form name and it works
  2. If no form ID but the form name has a space then the system makes a form_ID which has a space in it and then you cant submit
1 Like

There's been some discussion on the GitHub issue, and after more investigation, it seems like this is likely an issue with Enketo rather than with ODK Central: see https://github.com/enketo/enketo-express/issues/252. For now, probably the best way to avoid the issue will be to use the settings tab of the XLSForm to specify a form ID that does not contain a space.

1 Like