XML File Creation in ODK Collect

Support
1

What is the problem? Please be detailed.

S4W-Nepal recently organized a citizen science flow (CS Flow) campaign to map pre-monsoon flow rates of the different rivers in the Kathmandu Valley. We developed an ODK form for this that includes capturing a video of the salt breakthrough curve for the salt dilution flow measurement method. Since we ended up with roughly 30 GB of data, we decided to use Briefcase instead of loading data to our Aggregate server running on the Google Cloud.

To manage the data, we manually copied the ODK directory from the phones into a single computer and then loaded them into Briefcase. For 11 of the instances folders we noticed that there weren't any XML files but there were pictures and videos for the instance, and a folder had been created. The user is confident that all the forms were saved and finalized in the same manner.

The main question is what triggers the writing of the XML file into the odk/instances folder? Somehow we ended up with a folder and pictures/videos being created, but no XML file. We are guessing that somehow the odk/metadata/instances.db file somehow buffers the data when the form is being filled in Collect and that something triggers the final writing of the XML file, but we would like to understand the process better so we can avoid these losses of data in the future. Strangely, the instances.db file that we have in the ODK folder was last updated 2 days prior to the dates of the missing XML files.

What ODK tool and version are you using? And on what device and operating system version?

ODK Collect v1.14.1

What steps can we take to reproduce the problem?

We are also having trouble recreating the problem, but are hoping some technical insight will avoid future cases.

What you have you tried to fix the problem?

We have talked about re-creating the ODK instances again through ODK Collect, and then manually modifying the GPS coordinates in a text editor to correspond with the correct locations, but we are waiting on this as a last resort.

Anything else we should know or have? If you have a test form or screenshots or logs, attach here.

Screenshot with XML file (all is as it should be!)

image
image1332×722 141 KB

Screenshot with missing XML file

image
image1131×722 126 KB

Screenshot of odk/metadata folder with old date modified for instances.db etc. (earlier than several of the instances folders)

image
image938×126 11.5 KB

2

How did you copy the files from the phone to the computer? I ask because the device drivers for connecting to Android can be pretty buggy so it's not impossible that during the copy it dropped a file. I would use a file manager on the device and check the instances folder on that device to see if the XML is there.

3

Thanks for the response @yanokwa! We connected the phone over a USB cable to a computer using the device drivers. We will try what you suggest here.

Just for our understanding, what triggers the XML file to be written in Collect? Is it when you press the save button, or at the end of the form by clicking "Save Form and Exit", or only if "Mark form as finalized" is checked, or ??? Thanks.

4

Unfortunately, we were unable to recover the XML files from the phone. I've searched the documentation and can't find any answer to my question about when the XML files are actually first created in the instances folders. Could you @yanokwa or someone from your team provide any additional insights into what triggers this to happen? Thanks!

5

You'll have to look at the code. This is where the instance are first created.
https://github.com/opendatakit/collect/blob/master/collect_app/src/main/java/org/odk/collect/android/activities/FormEntryActivity.java#L2431

6

Great, thanks @yanokwa for the additional information.