ODK Collect v1.26 storage migration

Starting in ODK Collect v1.26, a banner will be displayed on the main screen to trigger a storage migration. For most people, migration should be safe, quick, and have no impact on data collection.

Organizations with custom Android applications or who directly pull data from or push data to Collect should read this post carefully before migrating. Feel free to ask clarifying questions in the comments.

Why do I need to migrate files?

Google recently started requiring that applications in the Play Store make compatibility updates with every Android release. They've also made a requirement for applications to use private data folders. Prior to v1.26, ODK Collect stores all of its files in a directory that will no longer be available to updated applications after August 2020.

We are requiring users to trigger this migration manually starting with v1.26 to give organizations time to adapt before it is required. We will trigger the migration automatically in the update that will be released in mid-July 2020.

What are the implications of this migration?

For most people, migration should be safe, quick, and have no impact on data collection. However, one change is that ODK form definitions and filled form submissions will be deleted when ODK Collect is uninstalled. Before the migration, files stayed on /sdcard/odk even if the application was deleted.

If your workflow requires you to manually access ODK files, you will find them at /sdcard/Android/data/org.odk.collect.android/files instead of /sdcard/odk.

You will no longer be able to use Android applications that directly access ODK files (see below).

If you're thinking about buying Android 11 devices (available fall 2020), note that they will not be able to run versions of ODK Collect prior to the fall 2020 release.

Is there any risk for data loss in the storage migration?

The storage migration needs to move all files managed by ODK Collect which may include unsent submissions. We strongly encourage you to send all of your submissions before migrating. If you can't do this for some reason, we believe that the risk of data loss due to migration is extremely low.

What if I use ODK Collect with an external Android application that accesses /sdcard/odk?

For now, you should not perform the migration.

We have started collecting requirements for new ways for Android applications to get the same information from ODK Collect. Please comment on Collect will need to stop using /sdcard/odk for files describing what problem your application solves and what information it needs from ODK Collect.

What happens if I don't migrate before the mid-July release?

If you don't manually trigger the upgrade and then get the mid-July 2020 release, files will automatically be migrated in that last update.

If you don't manually trigger the upgrade and then miss the mid-July 2020 update but get later updates, it will be impossible for ODK Collect to access your old files. It will still be possible to recover your files manually if needed (with a file browser Android app or adb).

6 Likes

Thanks a lot for the heads-up

1 Like

Thank you very much for the information, but is it affecting the data that stored in the ODK Aggregate server or database?

1 Like

No it does not affect anything on the server side.

1 Like

In our mother tongue literacy work flow we use one app to lead the user to install other apps if they wish to be a volunteer contributor of their mother tongue. This process leads them through the links to install ODK and RecForge2. At the same time we write the forms needed for entering their mother tongue or videoing their hand signs (if Deaf) to the ODK/Forms directory.

We have done it this way because we have found that uptake on new volunteers is hampered if it is too complicated to install many different apps. So yes, the www.HisHandsReader.org is depending on writing to the ODK/Forms directory.

With 1.5 Billion students out of school in developing countries and our trying to roll out curriculum and receive new language inputs to support these kid's needs this comes at a very bad time for changing how ODK stores data.

Since 99 percent of the 1.5 Billion children live in places that won't see Android 10+ devices for another 10 years wouldn't it be logical to break step with Google for the best interests of those in the developing world?

If it is not possible to have ODK write to the Downloads directory then shadow host a version of ODK on FDroid that continues to keep data in the same place.

Thanks so much for your help in this matter.
Bob Achgill

2 Likes

I have not looked at the new builds of ODK in awhile. It used to be the case that we had to use RecForge2 to capture audio inputs from the enumerator. Is that still the case or does ODK now have onboard audio capture through the phone microphone?

If ODK still depends on RecForge2 then will this new development from Google also break the integration between ODK and RecForge2?

It's great to hear that ODK has been helpful for literacy in this way!

The storage change is mandated by the Play Store so we have no control over the timing. We have tried to give as much advance notice as possible and are glad to see that you now have a couple of months to adapt. The banner to opt into the change has been in ODK Collect for two months. Did you learn about it from the community announcement? Do you follow the releases category and participate in the beta? I'm interested to know what we might do in the future to make sure everyone knows about these kinds of changes as early as possible.

Audio capture still relies on an external application in order to give users control over external microphones or audio quality enhancements they might need. However, we have sometimes heard interest in an embedded recorder. If this is something that would be useful to you, consider writing up a feature description. In the mean time, the integration with RecForge 2 or other applications continues to work.

It sounds like other features could help achieve what you need on install. For example, if all forms were downloaded as soon as a server were configured, that might do what you need. If you’d like more ideas for designing an install process for your volunteers, please write a new support post describing in detail what your scenario is (how do volunteers learn about the software needed? Where are the forms hosted? How is the appropriate language determined?) and how you’re addressing it now (what writes the form definitions to the sdcard?).

Hello hello

We have ODK distributed as an apk within an independent app (and then its hidden from the launcher) so any updates our clients get reach only when we update the apk we distribute (most clients who use the odk through our app have no idea its a completely independent app).

Odd question, if we simply do not update the apk that we distribute, are there any expected issues we should expect past 2020 if the users devices are not running Android 11 (minus the obvious drawbacks of running outdated apps)?

No you shouldn't run into any other inconveniences. The only thing I can think of is that once you decide to update your app in the future and keep the data stored on your devices you will need to perform the migration on that old version because once you install a version that uses only scoped storage that version won't have the migration option because it won't be able to do that either way (due to no access to sd card).

alright. cool, thank you very much.