ODK Collect Data Recovery- Data Lost

Support
1

The Issues:
Hi everyone! I am very new to ODK and ODK Collect which I have been using to work on a field project. Because of my naivety with ODK, I had a colleague help me to create the survey and set it up on some tablets. The data collection was done offline and lasted about a week. Now that I am back in the office I have been trying to download the data.

My understanding of ODK Collect is that survey responses are linked to a cloud-based aggregator, which is where the survey XLSForm and responses live. When collected offline they have to be manually sent to the aggregator. However, before uploading the responses, I accidentally updated ODK Collect which logged me out of my project.

After consulting my colleague, I came to find out that they manually downloaded the survey onto each tablet and did not have the project linked to an account (it was done on the demo server). So after being logged out, I have no idea how to get those responses back. I have no way of logging back into the project to my knowledge.

I've tried looking in the files on the tablet (Samsung) and I've tried hooking the tablet up to my computer and sifting through the files again but I can't find them anywhere. Do any of you have any tips on how to recover the responses or are they gone forever :frowning:

2

Hi @Rczumberg,

Welcome to the ODK community! Whenever you have a moment, please take the time to introduce yourself here. Adding a picture to your profile is a great way to boost engagement and help us get to know you better.

Don't worry, your data might not be lost.

First, connect each tablet to your computer using a USB cable and ensure the tablet is set to "File Transfer" mode. Navigate to the ODK directory on the tablet's internal storage and find the instances folder, which contains all the collected form data. Copy this entire folder to your computer.

Next, download and install ODK Briefcase. Open ODK Briefcase, go to the Pull tab, and select the folder where you copied the instances folder. Then you will be able to download the data.

By following these steps, you should be able to recover and download your data. If you encounter any issues, feel free to ask for further assistance. Good luck!

[ODK Briefcase Documentation] (https://docs.getodk.org/briefcase-intro/#odk-briefcase)

3

There is some great advice @Joseph_Muganga!

To add some texture...

When you have connected the device, you will need to access the folder [your-device-name]\Phone\Android\data\org.odk.collect.android\files\projects
The next folder down will be a 'uuid' (so not a meaningful name) and below that you will find 'instances' (don't panic, keeping digging!)

One thing that is worth noting @Rczumberg about Briefcase is that it reads the 'instances' you have downloaded into a database and things can get confusing when you download from more than one device. I would recommend taking one device at a time, download the instances to a local folder, then follow the steps for extracting them with Briefcase. When you have successfully exported those responses to csv, then try the next device.

You might find it necessary to reset the Briefcase database after each device - which is why it is good to export the data before trying the next device! If you've got lots of devices, it will be very upsetting to get the the penultimate one and for Briefcase to overload!

I'm speaking from some frustrating experience. But the good news is that I have never lost any data (respect, ODK!) - although I have more grey hair for my trouble :slight_smile:

An alternative would be to collate all of the instances from different devices into a single folder on your computer, but it depends whether you have a question in the form that will help you identify the enumerator (if that's not important, it might be the easy way to manage this process, but again it introduces some risks)

Good luck!

1 Like
4

Hello @Joseph_Muganga and @seewhy!

Thank you both very much for your helpful and thoughtful replies! I really appreciate the clarity of our responses. To provide a bit of closure, I am sad to report that the data is in fact gone :frowning: . When updating, I had deleted the app and redownloaded it before "sending completed forms". After carefully combing through the instances folder, using ODK briefcase and a number of data recovery applications, I was not able to find the files. You live and you learn I guess.

As a result, now I am petrified to try to extract the information from my remaining 3 tablets. The data collection process was extremely labor intensive and unfortunately cannot be replicated as it is for our pilot study.

As such, I would appreciate some more help on how to carefully extract the data from the remaining 3 tablets. Right now, all of my completed surveys are waiting in the "send completed surveys" tab in ODK Collect. I am afraid to send them because I am not sure where they will go considering I am not using a server (everything is locally). I assume they will go to the "instances" folder that you both mentioned- however, I was hoping to get some confirmation before I erase even more of our hard-earned data :sweat:

I also had a clarifying question about ODK Briefcase. I tried doing a test run using some old surveys collected on the tablet but I was stumped. Specifically on the "pull" tab I couldn't figure out where I am supposed to select "collect directory" or "form definition". I ruled out the two server options given that everything is being done locally. I tried using both form definition and collect directory though both gave me error messages.

As always, I truly appreciate your help!
Rachel

1 Like
5

Ah, yes, if you uninstalled the app that might have been terminal for the data on your device!

The instances folder will have a series of folders, one for each completed form (submission) - in each folder will be a.xml file and any media - if those folders are empty, you have no data.

So, on to the other 3 devices....
The data is stored in the instances folder and they don't 'go anywhere' unless you delete them from the device. If the tablets have not been set up with a Central or Kobo server connection then there is nowhere to send the submissions. But don't try out of curiosity until you've got it on your hard drive!

The instructions for briefcase should see you through. But just for clarity (and to include what I forgot previously) here is an example:

Create a folder on your PC, maybe called ODK.
Navigate to the uuid folder on the first tablet
You should see folders called forms AND instances (among others)
Copy BOTH of these to the folder ODK on your PC

Open ODK Briefcase and if you haven't already, select a (DIFFERENT) folder as the 'storage location' (under 'settings')- this will create a new folder on your PC called Briefcase Storage (with subfolders when you pull data)

On the 'Pull' menu, use the dropdown and select 'Collect Directory' and navigate to the ODK folder you created in step 1 - that will read the form definitions that are in the forms folder.
You should see the form listed, so select that and click 'Pull' (that basically copies everything to the Briefcase Storage folder you just created)

On the EXPORT menu you can then configure what to export and where...

It's been a while since I used Briefcase but it was my only way of transferring data for years and it's easy to use once you get the hang of it - pull the data from the ODK folder, Briefcase creates a copy of everything, then export to csv (and you get a 3rd copy of any images!) - the configuration of EXPORT is important as you might NOT want to overwrite existing files (in the destination folder), given how precious your data is.

When you are doing the next round of data collection I recommend that you use a server for submissions - Central is very robust (you can host your own) but other flavours are available.

Hope this helps...

6

@seewhy

Thank you so very much! The "forms" fold is what I was missing. Your instructions were very helpful and crystal clear- I was able to successfully download and export the rest of my data.

As for your suggestion about the server I completely agree. There was a miscommunication early on in the survey creation. I had thought that everything was set up with a server and aggregator but I came to find out that this was not the case. In light of this, I've set up a server on kobo and am all set to go for the next round of data collection.

Again, thank you very much for your help/patience with me as I navigate and learn ODK :slight_smile:

Cheers

1 Like
7

Well done.

Been there, lost the T-shirt!

The pain of potentially losing data (and actually losing it) is debilitating even though it's 'just data'. It is one of the reasons that I've stuck with ODK for all my field data collection - there's usually a way to recover what you were working on as long as you still have (access to) the device - sounds like you were pretty thorough at making sure your Samsung erased things though - good effort :wink:

And this community is very helpful and patient - they've had to put up with me for long enough :slight_smile: Welcome aboard.

3 Likes