ODK Collect storage migration

Support
1

Starting in ODK Collect v1.26, a banner will be displayed on the main screen to trigger a storage migration. For most people, migration should be safe, quick, and have no impact on data collection.

Organizations with custom Android applications or who directly pull data from or push data to Collect should read this post carefully before migrating. Feel free to ask clarifying questions in the comments.

Why do I need to migrate files?

Google recently started requiring that applications in the Play Store make compatibility updates with every Android release. They've also made a requirement for applications to use private data folders. Prior to v1.26, ODK Collect stores all of its files in a directory that will no longer be available to updated applications after November 2020.

We are requiring users to trigger this migration manually starting with v1.26 to give organizations time to adapt before it is required. We will trigger the migration automatically in v1.29.

What are the implications of this migration?

For most people, migration should be safe, quick, and have no impact on data collection. However, one change is that ODK form definitions and filled form submissions will be deleted when ODK Collect is uninstalled. Before the migration, files stayed on /sdcard/odk even if the application was deleted.

If your workflow requires you to manually access ODK files, you will find them at /sdcard/Android/data/org.odk.collect.android/files instead of /sdcard/odk.

You will no longer be able to use Android applications that directly access ODK files (see below).

If you're thinking about buying Android 11 devices (available fall 2020), note that they will not be able to run versions of ODK Collect prior to the v1.29 release.

Is there any risk for data loss in the storage migration?

The storage migration needs to move all files managed by ODK Collect which may include unsent submissions. We strongly encourage you to send all of your submissions before migrating. If you can't do this for some reason, we believe that the risk of data loss due to migration is extremely low.

What if I use ODK Collect with an external Android application that accesses /sdcard/odk?

For now, you should not perform the migration.

We have started collecting requirements for new ways for Android applications to get the same information from ODK Collect. Please comment on Collect will need to stop using /sdcard/odk for files describing what problem your application solves and what information it needs from ODK Collect.

What happens if I don't migrate before v1.29 is released?

If you don't manually trigger the upgrade and then get v1.29, files will automatically be migrated.

If you don't manually trigger the upgrade and then miss the v1.29 update but get later updates, it will be impossible for ODK Collect to access your old files. It will still be possible to recover your files manually if needed (with a file browser Android app or adb).

7 Likes
2

Thanks a lot for the heads-up

1 Like
3

Thank you very much for the information, but is it affecting the data that stored in the ODK Aggregate server or database?

3 Likes
4

No it does not affect anything on the server side.

1 Like
5

In our mother tongue literacy work flow we use one app to lead the user to install other apps if they wish to be a volunteer contributor of their mother tongue. This process leads them through the links to install ODK and RecForge2. At the same time we write the forms needed for entering their mother tongue or videoing their hand signs (if Deaf) to the ODK/Forms directory.

We have done it this way because we have found that uptake on new volunteers is hampered if it is too complicated to install many different apps. So yes, the www.HisHandsReader.org is depending on writing to the ODK/Forms directory.

With 1.5 Billion students out of school in developing countries and our trying to roll out curriculum and receive new language inputs to support these kid's needs this comes at a very bad time for changing how ODK stores data.

Since 99 percent of the 1.5 Billion children live in places that won't see Android 10+ devices for another 10 years wouldn't it be logical to break step with Google for the best interests of those in the developing world?

If it is not possible to have ODK write to the Downloads directory then shadow host a version of ODK on FDroid that continues to keep data in the same place.

Thanks so much for your help in this matter.
Bob Achgill

2 Likes
6

I have not looked at the new builds of ODK in awhile. It used to be the case that we had to use RecForge2 to capture audio inputs from the enumerator. Is that still the case or does ODK now have onboard audio capture through the phone microphone?

If ODK still depends on RecForge2 then will this new development from Google also break the integration between ODK and RecForge2?

1 Like
7

It's great to hear that ODK has been helpful for literacy in this way!

The storage change is mandated by the Play Store so we have no control over the timing. We have tried to give as much advance notice as possible and are glad to see that you now have a couple of months to adapt. The banner to opt into the change has been in ODK Collect for two months. Did you learn about it from the community announcement? Do you follow the releases category and participate in the beta? I'm interested to know what we might do in the future to make sure everyone knows about these kinds of changes as early as possible.

Audio capture still relies on an external application in order to give users control over external microphones or audio quality enhancements they might need. However, we have sometimes heard interest in an embedded recorder. If this is something that would be useful to you, consider writing up a feature description. In the mean time, the integration with RecForge 2 or other applications continues to work.

It sounds like other features could help achieve what you need on install. For example, if all forms were downloaded as soon as a server were configured, that might do what you need. If you’d like more ideas for designing an install process for your volunteers, please write a new support post describing in detail what your scenario is (how do volunteers learn about the software needed? Where are the forms hosted? How is the appropriate language determined?) and how you’re addressing it now (what writes the form definitions to the sdcard?).

8

Hello hello

We have ODK distributed as an apk within an independent app (and then its hidden from the launcher) so any updates our clients get reach only when we update the apk we distribute (most clients who use the odk through our app have no idea its a completely independent app).

Odd question, if we simply do not update the apk that we distribute, are there any expected issues we should expect past 2020 if the users devices are not running Android 11 (minus the obvious drawbacks of running outdated apps)?

9

No you shouldn't run into any other inconveniences. The only thing I can think of is that once you decide to update your app in the future and keep the data stored on your devices you will need to perform the migration on that old version because once you install a version that uses only scoped storage that version won't have the migration option because it won't be able to do that either way (due to no access to sd card).

11

alright. cool, thank you very much.

12

A post was split to a new topic: Wrong password entered

13

Hi,

Understanding the change from the internal side. However how will it effect the participants using ODK collect? What changes do the users need to be informed of? (From what I can understand is user must upload data before the latest update without any losses?)

We the organisation pull the data using briefcase from the android tablets in multiple sites. Some help in regards to this enquiry, would be great.

Thanks.

14

If users don't need to use external applications that integrate with ODK Collect, it should not change anything for them. We recommend submitting all data before the update but it's not strictly necessary.

We have updated the documentation at https://docs.getodk.org/briefcase-using/#collect-directory to describe this process. In particular, see https://docs.getodk.org/collect-adb/#collect-directory for the location of ODK data after the migration.

15

A post was split to a new topic: Failure message when attempting to submit

16

Hi @LN

How launching collect from external app will be affected? https://docs.getodk.org/launch-collect-from-app/

17

Nothing that is officially documented is affected so if apps only use features from that documentation page, they will continue working without problem. It should also be quick to verify by running the migration on a test device and trying out the integration. What will stop working is attempts to read from or write to the odk folder directly.

18

Thank you for this info.

1 Like
19

Hi friends, i not very familiar to Odk. But i want to collect data for my study using ODK collect. I have already created my questionnaire. But on the main screen of odk collect i have encountered a message asking me to migrate collect forms to private storage. I dont real know how to go about it and Iam worried about my data loss. Can i get a brief detail about this. Does it mean the form i have created in kobotoolbox is going to lose? Do both sent and unsent collected form going to lose. How can i migrate? What happen if i go for data collection without migrating a form i have created?

20

Hi @Kingdavid
welcome to the forum! Please introduce yourself here!

the migration should be safe however we advise users to upload already filled forms to make sure that your data is safe. you can also use Briefcase to make a copy of your data. Once you start the migration your data should be just copied to private data folder and you shouldn't lose anything.

21

Thank you for this information. My issue is, with this new storage, will it still be possible to access forms and manually retrieved data from broken tablets?