Collect will need to stop using /sdcard/odk for files

Development
,
1

Google recently introduced stricter target API level requirements for applications published to the Play Store. ODK Collect currently targets Android 9 but as soon as Android 11 is released, it will need to target Android 10. This will likely be in fall 2020.

This has two major implications: Collect will no longer be able to use files in /sdcard/odk/ and it will no longer be able to read device-unique identifiers such as the International Mobile Equipment Identity which is used as deviceId (sent with every form submission and form list request). Because these may have impact across the ecosystem, I'll start a thread for each change describing a proposed approach and soliciting feedback. See Collect will need to stop using IMEI as deviceID and making simSerial and subscriberID available for the thread on deviceId.

The /sdcard/odk/ folder that has always been used by Collect to store blank forms, submissions and corresponding databases in a public folder on the device that any application can access. For security purposes, Android will no longer allow folders shared between applications. Collect will instead need to use a folder that only it can see and write to ("scoped storage").

As far as I know, these are the implications:

  • When ODK Collect is uninstalled or downgraded, all files including unsent submissions will be deleted
  • If users want to manually push files from a computer to their device, they will need to use a path like /sdcard/Android/data/org.odk.collect.android/files
  • It will no longer be possible for other applications to directly read or modify ODK Collect files (e.g. generate or modify submissions, read form definitions)
  • It will be impossible to use ODK Collect versions prior to the ones with this storage change on Android 11+ devices

@Grzesiek2010, @seadowg and I have spent quite a bit of time exploring our options and would like to propose the following course of action:

  • Let users know that these changes are coming with a community announcement
  • In the upcoming v1.26 release, have new installs use the dedicated Collect directory. This means that these users will not need to undergo any kind of data migration.
  • Either in v1.26 or in the following release, put a notice on the Collect landing screen that files need to be moved and let the enumerator make the change by tapping a button when it is appropriate for them (e.g. when all pending submissions have been sent). We are proposing to let users explicitly trigger the data migration because it could take a while depending on storage speed, whether sent submissions were deleted and whether media files were captured. Having it be user-triggered will also make it easier to recover from any errors that may happen during the migration.
  • In the summer release (before August 2020 and the requirement to target Android 11), on devices where the files haven't already been moved, force the move on first load of this version. After the change to scoped storage is made, it will be impossible for Collect to migrate files out of /sdcard/odk so users who upgrade after will not have blank or filled forms available. They should be able to manually move the files using adb, however.

Please provide any feedback you may have on this approach. Are there implications we may not have considered? Do you have alternate suggestions on how we make this change as seamless as possible for users?

OpenMapKit is one application that I know reads contents from /sdcard/odk and will need to change (@danbjoseph). The Skunkworks Crow project for sharing form definitions and filled forms between devices will need to make a similar modification for itself. I'm not sure what the best approach will be for reading and writing form definition and submission XML but I think it should be possible using FileProvider functionality. @Shobhit_Agarwal, @dexter21, @Mickys0918 is this something you've explored? I think it will require changes to Collect. Do we know of other Android applications that do things explicitly with the /sdcard/odk directory? @Jason_Rogena (Ona)?

CC @tomsmyth (NEMO), @Ukang_a_Dickson (Ona), @jnm (Kobo)

3 Likes
2

Thanks for flagging this @LN. I'll use this as a reason to loop back around with various people who do OSM data collection (especially Humanitarian OpenStreetMap Team folks) and restart conversations around the future of OSM mobile data collection. Maybe we can find/pool resources to continue @zestyping's great work (with the geo widgets and more recently the 'viewing filled forms on a map' functionality) instead of further investment in the OMK Android app.

1 Like
3

it will be /storage/emulated/0/Android/data/org.odk.collect.android/files

1 Like
4

You are correct that this is likely to be the actual path. /sdcard/ is a symbolic link to /storage/self/primary which itself links to the current user's primary storage (e.g. /mnt/user/0/primary) which is itself a link into emulated storage. There's more info in this Android StackExchange post. I believe we'll want to document the /sdcard prefix because we can't know the exact actual path. For example, if there are multiple users and a non-default one is logged in, the actual path prefix would be something like /storage/emulated/1/.

1 Like
5

I don't think we should force new installs to use the dedicated directory. We don't know for certain if new users are less likely to use apps that read the odk folder. Given that the migration is not yet necessary, we shouldn't force it. I'd rather new users see the opt-in screen, get all the necessary context, and decide then.

Everything else about the proposed course of action is, as always, thoughtfully done. In particular, I think the early opt-in is nice because it provides lots of warning about the upcoming change very early. I don't necessary think that enumerators are best positioned to decide if they should do this migration, but we can start by defaulting to not migrating, explaining the timeline, adding language in the dialog for them to speak with supervisors, etc.

Thanks for your leadership on this, @LN.

6

I think we should use a banner for this:

image
image1520×760 183 KB

If we want to inform users about upcoming changes first we can have just LEARN MORE option and then add FIX IT in the future (but personally I'm not sure whether we should separate it or not).
I think a banner is better because dialogs might be ignored by users, @ln you have mentioned a button in our conversation but it would be ugly as you noticed.

As the doc says banners are for: prominent message and related optional actions so I think it's what we need.

1 Like
7

A banner on the Main Menu seems a like good call to me :+1:

8

Ah right yeah that makes sense. I guess it's always going to have to be optional.

9

Hi @LN , does it mean that the sdcard will not be used anymore ? Or does it "only" mean that the path will change on the sdcard ?
I have in mind the necessity to put big mbtiles on the sdcard when internal space is too small...

10

The /sdcard/ path is a bit misleading. It will use the sdcard if one exists and shared internal storage if not. That aspect of the behavior will not change. So yes, from an external perspective, it's "only" changing the path on the sdcard.

Are you currently using physical sdcards? Have you had any issues with users taking them out?

1 Like
11

In fact not now because it is not allowed on our phones/android version but it was a possibility on older and smaller phones.

1 Like
12

We've been making good progress on the migration implementation.

@danbjoseph thanks for your response and for the enthusiasm around getting some of the current OpenMapKit functionality into Collect.

Thanks. I've circled back with @Grzesiek2010 and @seadowg and we've agreed not to have any special behavior for new installs. Everyone will need to opt-in.

Thanks. I am working on some UI options to share soon. I'll share them in the Collect Slack first so that @Grzesiek2010 and @seadowg you can provide a first round of feedback.

I've also put some thought into designing an official API for external apps to supply form definitions and form submissions to Collect or to get XML form definitions or submissions from Collect. Some questions I am wrestling with and am interested in feedback about:

  • should we immediately replace existing access to form and submission XML? We could say e.g. we don't have clear demand for a replacement API since we don't yet have examples of actively deployed apps that are requesting this continued support. (I lean towards providing a replacement API but it's not entirely trivial and it does mean another integration point to support moving forward)
  • if we do, whether this replacement needs to be announced and perhaps released BEFORE users start having file migrations so that folks with app integrations can react
  • should we bump the version to Collect 2.0 when we start rolling out the file migration? For developers, this will make it very clear that there are breaking changes. For most users, this would seem confusing because there won't be major visible changes. Since folks generally auto-upgrade through the Play Store, the more significant version change probably wouldn't make a difference for most.

I have identified two other applications with integrations that will be affected:

  • OpenHDS @aurdipas do you know of active deployments? The only one I am aware of is the one @Batkinson works on but they have now forked both the OpenHDS mobile client and Collect. @Batkinson do you expect to stay on your forks indefinitely? Do you expect to address the scoped storage requirement or deliver outside of the Play Store once it is enforced?
  • Project Buendia I've been talking to @zestyping and he has agreed to review a new API. The project uses a Collect fork currently so would not be immediately affected.

Both are examples of applications that improve longitudinal data workflows and add communication with some custom source of entity (patient) data. I'm certain there are many other such applications out there but the big questions is whether they are in active use and would leverage a new API if one were introduced.

@TSC-1, I think this is something you will likely want to discuss during your next meeting.

13

Please take note @adam.butler (next facilitator)

14

OpenHDS is the latest development and it is used in (at least) 7 HDSS sites.

2 Likes
15

Ahhh, of course, I should have looked at the Swiss TPH Github. Thank you for that information. In that case I think we do need to provide an alternate API as soon as possible. Is the OpenHDS app deployed through the Play Store?

1 Like
16

Just to make sure it nothing slip thru cracks, we should test last-saved doesn't break (since it's retrieving previous submission from here, right @ln?)

1 Like
17

This Google doc has a proposed user experience. Please comment directly inline in the doc if you have feedback. I'm particularly eager to hear ideas for simplifying things for enumerators.

I have also shared that doc and a summary in a features thread at Let users know of required storage location change and let them opt-in before August 2020 in the hopes of getting feedback from folks who don't watch the development category.

Yes, that's correct. :pray:

18

nope. Is not on the playstore. And is not maintained/developed anymore due to lack of funding. As far as I know it works max with Android 7 and we suggested to openHDS users to don't update anymore ODK Collect to avoid issues.

2 Likes
19

Apologies for the delayed response. I couldn't get to this until now due to competing priorities.

TLDR: The forks aren't going away. We likely will address the scoped storage requirement, but not necessarily the same way as mainline Collect.

What meaning does the "now" in "now forked" have? The fork of openhds tablet happened circa-2015 and the fork of collect only just happened within the last few months. They are also very different in nature.

After working with original team from the University of Southern Maine, I eventually took over development of their existing fork of openhds for the same project on Bioko Island. I just "production"-ized it by giving it a home, branding it, and fixing its major flaws to function for its intended purpose (which was not the same as openhds). There wasn't an active online openhds development community to integrate back to, and many of the changes that needed to happen were not clear conceptual fits with a demographic surveillance system. To my knowledge, it is the only actively developed fork of that project, including the original.

The collect fork happened for different reasons. Mainly, instability of updates from Google Play kept breaking our daily operation of around 50-100 devices. However, I had been reluctantly considering the need prior to that, for similar reasons to the openhds fork: its purpose is not really the same as collect. Many features that would not make sense in Collect (as it exists today) make perfect sense for the fork. For that reason I don't see it going away, at least until the project does. However, we want to track as closely with mainline as possible, both to keep maintenance reasonable, as well as to continue to benefit from all of the great work the ODK community is doing.

Yes, we will likely be adressing the scoped storage requirement for CIMS Forms within the next couple of weeks. We are interested in a Collect solution, but don't expect direct support of any kind (including expedited delivery of an api) from the ODK community. This will likely be bundled with changes to way the apps integrate, since the Collect model had some undesirable quirks that we will want to address.

I can't speak to Project Buendia, but we are not solely tracking patients. We actually track households and inhabitants on Bioko Island for accurate coverage reporting and other metrics. We also have plans on expanding that support to track arbitrary entities, since different organizational units want to leverage the same shared database during inital data collection for easier analytics downstream. For example, tracking bednets, informal drug dispensaries, private clinics, etc.

Happy to expand on or clarify any of this if anyone is interested.

20

Thanks for the context, @Batkinson. If you're up for it, I propose we hop on a quick call to discuss the scoped storage and app integration changes we're both planning. It's likely we'll have to do at least some of the same work so I think it would be beneficial to coordinate a little bit, especially since you hope to track the Collect mainline. I'll follow up with you directly and I can report back on this thread.

In the mean time, I have also checked in with @Jason_Rogena and @Kigamba at Ona and they have confirmed that they continue to maintain app integrations through the /sdcard/odk/ folder.