ODK Central upload error: A submission already exists with this ID, but with different XML

Support
,
1

Hello,

we have had (in a couple of cases already) a problem when submitting forms from ODK collect 1.30.1 to ODK Central 1.1.2.

The forms should be finalized (and submitted) automatically, but sometimes this does not happen immediately (due to connection), and when the enumerator tries to force the submission, he/she got the error message:

Generic Exception: Error: A submission already exists with this ID, but with different XML. Resubmissions to attach additional multimedia must resubmit an identical xml_submission_file.

It seems to be related to this this post but it has a slightly different error message.

One possible explanation (that I can think of) is that the submission process was somehow interrupted during the upload, so that ODK Central received some metadata, but not the complete form. Is it possible?

Thank you!

2

Here's what is happening...

The server is successfully receiving the submission, but the mobile gets confused and thinks the submission has failed. This usually happens on very poor connections.

This is usually not a problem because the mobile will eventually retry the failed submission, and once the server confirms receipt, the mobile will mark the submission as successful.

The problem with this campaign seems to be that when the submission fails, data collectors are opening or editing the failed submission.

Once they do that, the data inside the submission on the mobile will not match the data inside the submission on the server, so the server will reject the mobile's submission as invalid.

These are the error messages the data collectors are seeing. The best current fix for this issue is:

  1. Tell the data collectors that if a submission fails with this specific error, they should ignore it. The mobile will retry eventually and all will be well. They should NOT open or edit these failed submissions because it will change the submission and make it very difficult for us to do step #2.

  2. At the end of the campaign, gather all the phones with this problem and make certain that all failed submissions are indeed on the server. In unlikely case that they are not, you can manually upload them at that time.

3

Thank you @yanokwa, this might be the case (for sure we have already seen that connection in some cases is poor).
What I am not sure (but I will check from our side) is whether the enumerators can edit the form (completed, but not submitted), even if the QR code setting are thought to mark forms as finalized automatically.

Thanks again.

4

As long as forms are not sent, they can be edited. To fix the issue that you've run into, we'll likely change this behavior to prevent editing once a submission has been attempted. The issue has been filed at https://github.com/getodk/collect/issues/4589 and is under consideration.

5

Hi Yaw and team, I've found this forum thread because this issue has occurred a few times this week. The above explanation seems plausible enough and doesn't seem like much cause for concern, but I was wondering why the issue might be appearing now given the github link above suggests this issue was resolved in July 2021.
Many thanks, Michael

1 Like
6

This should have been resolved in July 2021, but yours is the second report we've received in the last few days. Please let me know the following?

  1. What version of Collect are you running (e.g., v2022.3.x)?

  2. Have the data collectors perhaps been hitting the Cancel button while the forms are being submitted?

  3. Has the Collect app been slow or crashing before or during submissions?

  4. Has the Internet connection been particularly slow or unreliable when you've been sending data?

  5. Any other unusual or unexpected behavior when submitting data?

7

Hi Yaw, I've passed this back to the data collectors in Zimbabwe and will let you know what they say.

1 Like
8

The enumerators have said the problem occurred when they were trying to edit and submit a form already uploaded, but they have not had any other issues.

I'm having trouble confirming the version of the Collect app they are using, but I'll reply again when I get that info.

1 Like
9

Very interesting! Our intent is for filled forms that have been sent to be impossible to edit from Collect. Do you know how they're making the edit? Is it through "Edit Saved Form"? If so, what is the status of the form when they make the edit (e.g. "Finalized" or "Error sending")? Thanks so much for helping us troubleshoot this.

10

We just ran into that error with a Samsung Tab A 8" 2019 running ODK Collect latest beta (at time of writing, Dec 2022) sending data from a remote island with a very unreliable mobile reception. We send forms with sometimes ca 2MB photos attached.

11

Thanks for reporting this, @Florian_May. Are you confident it was v2022.4 Beta 2 released Nov 16? If so, is it possible the filled form was originally saved using an older version and then was edited using Beta 2?

Starting in that version, the sending process starts by changing a submission's status to "failed" before even making any network requests. That should mean that there's truly no way to further edit that submission.

To troubleshoot further we'd need the submission from the device and the submission on the server to compare. If it seems possible to get those, I can provide instructions on how to do so.

12

Thanks for the instructions @ln!

I cannot tell the app version as the device currently sits on a remote island with limited technical expertise. All our devices run the Beta releases and have regular enough internet access for daily submission upload, so I'd expect them to auto-update to the latest Collect versions.

I'm awaiting the submissions once the expedition returns from the island. I'll grab a snapshot of the submissions from the server.

I'll post here once I know more.