Data loss scare when updating to a new form version

Came across some unexpected behavior in central today and wanted to note it so that other people can avoid it in future/troubleshoot it! Not sure if it is actually permanently fixable.

1. What is the issue? Please be detailed.
I have a form in central with ~4,000 submissions. Yesterday, I published an update to that form which nested a number of the questions inside that form within a new group, which appeared or did not appear based on a relevant condition. After doing so, the data from those questions in all prior submissions disappeared from Central, both from .csv downloads and viewing individual submission instances in enketo (I did not check pulling through odata).

Fortunately, I do mean disappeared, not deleted, as by rolling the form version back to one without that group, all of that data was recovered and was re-accessible in the expected way. Since none of this data was actually deleted, I'm assuming this issue was caused by Central deciding that these now nested questions were actually new questions entirely, and that the old questions had been 'deleted' from the new form version, and so as a result, it did not show them, like it usually shows removed questions from older versions.

From Central's perspective, I guess there may not be a great way to determine a question being newly nested vs simply deleted, so this may not be a bug per se and just something that will happen. That said, a warning of some sort to users when they upload a new form version that removes a bunch of questions or creates new versions of existing questions just with a different group prefix that the old data will no longer be accessible in the expected way may help make this a little more transparent, especially to newer/less experienced users. I could definitely see someone newer to ODK panicking if this happened and thinking that they lost a lot of their submission data permanently, but perhaps my intuition leans the wrong way here and most will anticipate this behavior.

Regardless, if anyway else does have this issue down the road, hopefully this shows up on google and is helpful!

For replication, if necessary:
central version is 1.5.4

1 Like

I'm sorry that we gave you a data loss scare.

When you export, you can include fields that are no longer in the published version of the form.

Setting relevance to false() is also a good solution. Both of these are documented at https://docs.getodk.org/central-forms/#updating-forms-to-a-new-version.

We can do better here. We should warn users when they remove or rename questions and explain how to get that data back into the export.

1 Like

No worries! Was able to fix the issue without to much trouble, but figured I'd toss it up here as nothing came up on an immediate google and hoped that others may find this useful. Realized as I was typing it up that it was less of a bug report and more of an awareness post.

Cheers!

1 Like