Re-creating identically named form / version after deletion on ODK Central

1. What is the issue? Please be detailed.

On our ODK Central server there was a form which we'll call abc. Due to problems with exporting data from abc due to data size (I won't go into details here), we migrated all of the submissions within abc to distinct new forms (abc1, abc2, abc3, etc.). This involved manually changing their form ID and using briefcase (along with some manual moves between folders) to move from one form to another. We then deleted the abc form definition on the ODK Central server. At this point in the process, all data which had previously been submitted to abc was now safely submitted to abc1, or abc2, or abc3, etc.

In order to account for the fact that some fieldworkers were out in the filed, disconnected from internet, etc. during this process, after deleting abc from the server, we re-created a form (identically) named abc. The intention was for this form to receive submissions from tablets which were configured prior to the migration described in the first paragraph.

However, in this process, we have discovered some issues which suggest that - despite having deleted abc from the ODK Central web application - there are remnants of abc still floating around and causing problems. This became apparent to us because:

  1. We were unable to upload to ODK Central a new abc form definition with the same version number as any of the previous (deleted!) abc version numbers. For example, when we tried to upload abc with version number 123 on ODK Central, we received a message saying "The version name you've specified conflicts with a past version of this form. Please change it to something new and try again."
  2. We are unable to submit form abc with version 123 via ODK Collect. Upon attempt, a window pops up saying " - Error: Generic Exception: Error: Could not find the resource you were looking for."
  3. We are unable to submit form abc with version 123 via ODK Briefcase. Upon attempt, we receive messages saying "Error sending submission...: This submission was created for a form version that is not on the server. You can manually upload older form versions or let this push attempt complete and then try again."

The most serious part of this issue is the fact that there are many hundreds of submissions form form abc with version 123 on dozens of android devices. We need to ensure that we safely extract those submissions from device to server/storage, but it's not clear how best to do this given the above 3 oddities.

2. What steps can we take to reproduce this issue?

Create a form with ID abc and version number 123 on ODK Central. Submit several times. Fill out several submissions to that form on an android device but do not submit. Delete the form on the server. Try to re-create another form named abc with version 123; you will fail (bullet point 2 above). Create an incremented version (abc, 124). Try to submit the submissions from the android device; you will fail (bullet point 3 above).

3. What have you tried to fix the issue?

We have tried several measures, but none are satisfactory. We can

Help

We would appreciate any guidance regarding how to navigate this issue. I suspect that there is a way to more "fully" delete form with ID abc and then re-create it form scratch (including the re-creationg of the version number of currently on-device submissions), but this likely involves manual modification of the postgresql database (which we're hesitant to do, given risks for data integrity).

Dear @joebrew,

when you delete a form in Central, this is a soft form deletion.
The form is marked as deleted in the form table in the database (the field forms.deletedAt has a not-null value).
So you can not upload a form with same formId and version.
If you want to make a Hard delete of the form to be able to then re-use the same formId/version, you would need to do it accessing the postgres db and deleting the record from the form table.
If you don't have knowledge of database management (especially inside docker), it is not really recommended to directly do changes in the DB.
You could mess up your installation.

If you can access your Central installation via terminal, then once you are in your central installation directory, to access the postgres db you need to execute the following command

docker exec -it central_postgres_1 psql -U odk -W

the default password is odk

the command \dt gives you the tables list in the database.

you then need to delete the record you want from the forms table.

I suggest first to check the list of forms on your installation that were deleted by running

select * from forms where "deletedAt" is not null;

@joebrew Everything but the data export issue you described is working as expected. It'd be good to get to the root cause of that issue, so perhaps you can write another support post about that.

@aurdipas is correct that you need to purge the deleted form, but I would suggest a different set of steps so you don't risk any corruption of your database.

  1. Log into the database as was suggested. The default password is odk

    docker exec -it central_postgres_1 psql -U odk -W
    
  2. Run the following command to get the form ID. Note that I've put the project ID in the SQL query. That ID is the URL of your server (e.g., https://example.getodk.cloud/#/projects/1).

    select id,"xmlFormId","deletedAt" from forms where "deletedAt" is not null and "projectId"='1';
    

    The first column is the id you'll need for the purge. xmlFormID is the identifier you've been referring to (e.g., abc). Use the deletedAt value to confirm you are getting the id for the form you want to purge.

  3. Now, exit from the database.

    \q
    
  4. Now force the form to delete. Replace YOUR_ID with the ID you got from the SQL query.

    docker exec -it central_service_1 /usr/local/bin/node lib/bin/purge-forms.js -f -i YOUR_ID
    

Thank you very much, @aurdipas and @yanokwa. Very clear.

@yanokwa, as per your suggestion, I've opened another support post for the data export issue: ODK Central submissions zip export fails with large dataset

1 Like