Provide a way to get mbtiles to Collect without having to connect to a computer

Ideas
1

Greetings ODK Community,
As a former Android developer and current Technical Project Manager at a software company in Nepal, Naxa, specializing in extensive geo-data work, I wanted to share an exciting proposed solution to address a limitation in ODK Collect regarding custom map layers. I’ve created a basic implementation of the idea to make it clear how it might work and to demonstrate proof of concept.

What high-level problem are you trying to solve?
ODK Collect has proven to be a valuable tool for our field surveying activities. However, we encountered a specific challenge when it came to transferring offline map imagery layers; since recent versions of Android make it difficult to place files in application directories, the only practical method is using a USB cable and external PC. Not everyone has access to PCs or the knowledge and time required to transfer offline files via USB transfer, and even for those able to do so it’s a hassle in the field. Faulty or malfunctioning cables often disrupted our data collection process, causing unnecessary delays and frustration. Many users, including ourselves, are glad to have the ability to use offline imagery/basemaps but wish it were easier to get and provide them.

Additionally, during the ODK Summit, a desire for offline basemaps was expressed by many stakeholders. While we don’t expect this to be a comprehensive solution, we think it could be an incremental improvement that will already help a number of users.

Any ideas on how ODK could help you solve it?

Introducing a Proposed Solution

To address this issue, I have implemented a feature in ODK Collect that enables the direct download of map layers from a URL into the appropriate application directory. This feature allows users to bypass the need for a USB cable and transfer the map layers seamlessly. Project managers can provide basemap imagery layers to their users and instruct their enumerators to download those map layers. It doesn’t disrupt or change the workflow of anyone already using the app; it’s an extra option available only within the basemap section of the project settings dialogue, and would likely only be noticed by users already interested in basemaps.

I have made an APK available on GitHub so people can test it out a version of Collect with this feature (requires uninstallation of ODK Collect while testing). It can be accessed through the following link [https://github.com/konishon/collect/releases/tag/v2023.2-mbtiles.0]. If this is of interest to the ODK community, I’d be glad to submit this as a pull request and go through the necessary reviews and process to integrate this into ODK Collect for everyone.

Using the Solution

To utilize this feature, follow the steps outlined below:

  1. Open your project in ODK Collect.
  2. Go to the project settings.
  3. Click to the maps list item.
  4. In the URL text input, enter the URL of your .mbtiles file.
  5. Wait for the file to download.
  6. Select your desired layer from the "Layer data file" option.
  7. Use this layer in your form.

Known Issues in the Current Solution

While the current solution addresses the USB cable limitation, there are a few known issues that should be taken into consideration:

  • Accessibility of .mbtiles: The .mbtiles file you are trying to download should be publicly accessible on the internet.
  • File Extension: The downloaded file should have the .mbtiles extension.
  • Overwriting Files: If a file with the same name is downloaded, it will replace the original file.
  • No support Big files download: Right now the download only work reliability for small files (less than 100MB)

Future Plans

In the future, I plan to enhance and improve this solution further. Some potential areas for improvement include:

  • Adding the option to include an MBTiles file in the QR code defining a project, as well as settings to determine if the file is automatically downloaded when the project is loaded into the phone (with the usual precautions such as asking for user permission and/or waiting for wifi before downloading in case of a low-bandwidth connection).
  • Refining the UX/UI.
  • Adding support to an authenticated file download
  • Enhancing the error handling and validation processes to provide better user feedback.

I am open to feedback and suggestions regarding the need for such a feature in the ODK app.

Upload any helpful links, sketches, and videos.

8 Likes
2

Hi @Aly_Blenkin @LN and @yanokwa! Bumping this thread.

Aly, this is the fix I mentioned on our call just now; since Android no longer permits us to place a file (like a basemap) in the relevant directory without a PC and USB connection, it's gotten a lot harder to use MBTile basemaps.

It would be great if we could simply provide a URL for a downloadable MBTile basemap, and have ODK (which obviously has access to its own directories on Android) to stuff that file in the right place.

This doesn't solve the whole problem of basemaps, but it does offer a fairly straightforward path for people to use specifically prepared basemaps for their projects.

@Nishon is our colleague in Nepal and has already shown in proof-of-concept that it's pretty straightforward, and we'd love to either submit it as a pull request or have your team work on it, whichever is easier for you (we'd be delighted to try to do the work, but we realize that this might not actually be the lowest-hassle route for you).

3

Thanks for bumping @Ivangayton, and for sharing, @Nishon! This came in just as we were deciding to take a first step in this direction for not the upcoming Collect release but the following one.

Our preliminary plan was to do something more along the lines of Add offline map layers without having to put files in Collect directory and Button to select an offline map file on the phone and move it at the right place. Having the user select a file rather than specifying a URL makes it possible to use device-to-device transfer through WiFi Direct, Bluetooth, etc.

We also discussed making it possible to specify an mbtiles file in form design so that Central would prompt for it and Collect would put it in the right place. @seewhy also proposed this in Send offline map to device from Central (we're not ignoring you, I promise).

We saw your post come in and haven't yet had a chance to consider the tradeoffs between that approach and others we've considered.

Are there special user needs that led you in the direction of specifying a URL in settings?

1 Like
4

How does the single URL implementation extend to having multiple tilesets? I have some single elevation sites with different tilesets for different uses (varying overlays or one is aerial photography and another is a layout drawing) and some multi elevation sites where each tileset is an elevation level.

This is much closer to what I was envisaging, tiles are added with other media, CSVs and downloaded.

Would these be stored at the project level like entities and only uploaded to Central once?

If uploaded to each form's attachments (currently any tileset is available to any form), if two forms in project A and one form in project B all call for the same tileset, will it download it to the device 3x and overwrite it twice? Or will there be a form layer folder like with images/CSVs and we end up with many copies over definition versions and different forms?

If the form can call for tileset.mbtiles as required, can it also set the current active tile layer based on a field in the form like ${active_tileset}? eg, some forms specify a tileset in the settings tab, but others either choose through a select_one, or, based on other selections, do an instance lookup to retrieve the tileset filename, and then that field is the tileset that is active in the map view.

2 Likes
5

I haven't looked at this implementation yet but I think the idea would be that it's just a download tool. Whatever URL you put there, it will download it, and you can use it over and over again.

Likely not initially. But Central doesn't store the exact same blob multiple times so although you'd upload them multiple times, you wouldn't take up additional storage.

No, similarly to Central, Collect detects when it already has an asset and doesn't download it again (which is missing from Enketo online forms currently).

How did I not predict you were going to ask for this?! Probably not initially but we would be working towards something similarly dynamic.

6

I'm very predictable!

image
image970×744 76.8 KB

I'm also very excited to see where this heads, having to sideload tilesets, or guide users into sideloading is non ideal.

1 Like
7

Hi to all and thanks for sharing your thoughts !

As Andrew, our definition of a project is more based on people using it than on the extend of the studied area.
Many forms are used in different small places that are far from their neighbor. For example we plan to map 70 farms in our whole region (more than 50 km between farms). We set one only form in the project, used by 10 enumerators who actually download and place the offline maps of the farm they plan to visit, where it has to be on the phone
And all the forms of the projects will not need the same base map (sometimes just a satellite view, sometimes satellite view with drawn objects on top of it...)

The tileset may be defined by the area name and the protocol used on the ground.
So I am also a fan of Andrew's gif :slight_smile:

2 Likes
8

Regarding the current implementation, as you described it perfectly, It functions as a simple download tool, fetching the file from the specified URL, and automatically placing it in the correct directory.

In terms of user needs, the challenges of dealing with Android's file system and instructing field enumerators to place the file correctly have been significant hurdles. By modifying Collect, we can simplify this process and make it more user-friendly for everyone involved. Additionally, allowing users to host map files separately. Also, adding support to download maps layers in collect looked more straightforward.

For, supporting multiple layers. For this current implementation
Instead of sticking to MBTiles, we can opt for a more versatile approach by using ZIP map packages. This way, users can bundle multiple layers together, and these layers could have different formats, such as KML

9

What do you think of the filepicker concept? I think it would address this because it would show recently accessed files and a small number of directories including Downloads. Here are how I imagine the two different workflows:

Downloader File picker
- Project manager sends URL via WhatsApp or in form field
- Data collector copies URL
- Data collector navigates to settings
- Data collector pastes URL
- Collect downloads mbtiles
- Data collector returns to question
- Data collector opens layer selector
- Data collector selects layer		 - Project manager sends URL via WhatsApp or in form field
- Data collector taps on URL
- Browser downloads mbtiles
- Data collector navigates to question
- Data collector opens layer selector
- Data collector browses to Downloads, selects layer

Did I capture the workflow you imagine with the downloader correctly or do you see it being used differently?

How important is this? Do you think being able to attach mbtiles to forms in Central like you can attach jpgs, csvs, etc would meet your needs as well? Note that for now all data collectors would get the same files. However, you would be able to specify a basemap for each question in your form individually (so different questions could have different basemaps).

1 Like
10

Sooooo happy to see this convo!

To be clear, I think of the idea of allowing a download of an MBTile file as a temporary fix. Using MBTiles has been made difficult by Android blocking access to the directory from which MBTiles are read, so people need a computer and USB cable to do something that should be possible without either (just download a damned MBTile).

There are a few ways to mitigate this quickly, one of which @Nishon has proposed (Collect itself handles the download into it's own directory) and another is @LN's suggestion of a file picker (Collect reads an MBTiles file from a directory the user can populate instead of only being able to see those in its own internal directory).

Either way is fine for me. Nishon's "download within Collect" is probably a bit more straightforward for users, and LN's "Collect uses MBTiles from other directories on the phone including 'Downloads'" is probably more flexible/powerful. I'd be happy with either for now; both would immediately allow most of the people I know and work with to use MBTiles basemaps relatively easily.

However, the "real" long-term solution, I think, is neither of these. The "right way" in my opinion is for the Collect app to deal with Tile Mill Servers (TMS) directly. That requires more development work and probably can't be implemented as quickly. The basic idea is supplying an Area of Interest and TMS URL. Collect handles the tile downloading (pre-caching) for the relevant area(s) directly from the TMS.

  • Project Manager sends Area of Interest (AOI) plus one or more TMS URLs via message or form field.
    • AOI is GeoJSON or Well-Known Text describing one or more polygonal areas.
    • Potentially more than one TMS URLs because as @ahblake points out, maybe you want more than one layer
  • Data collector is prompted "Do you want these tiles, which will cost you approximately X amount of data and local storage?"
  • If yes, Collect uses a geo-library to determine which actual tiles are needed to cover the AOI from the TMSs, and downloads them.
  • Collect map views use the tiles as background layers, user has layer selection widget to choose what basemap they want to see
  • Collect implements a few policy options for life-cycle management (how long to keep the tiles associated with a given form/project to keep the phone storage from filling up).
    • Tiles deleted when associated form deleted
    • Tiles deleted after a time-out period of some days
    • Oldest tiles deleted when directory exceeds some threshold
    • Other?

Two huge advantages to this approach:

  1. No need to store massive files on Central or any other server operated by the project manager. The project manager can, if they like and have the knowledge to do so, set up a TMS, but they don't have to. They can just store AOI polygons and point to whatever existing TMS they like (Bing satellite, for example, which is legally available for free use with OpenStreetMap mapping) for the enumerators' Collect apps to download.
  2. More potential flexibility for the enumerator to get just the data they need when they need it.
11

Can you elaborate on what makes it easier? My experience with copying URLs is that it can be pretty frustrating and error-prone. Maybe it depends on where the URL is coming from? How do you imagine it would be communicated to users?

What do you think of using the form design to specify mbtiles so Central can serve them (more detail at Provide a way to get mbtiles to Collect without having to connect to a computer - #3 by LN)?

If you don’t mind, I think it’d be helpful to split out the idea around specifying a tile server and linking to it. I also renamed this thread to capture the general pain point we want to address in the short term: mbtiles are already supported but they’re hard to get on devices.

12

Hi LN!

What's easier

I'm not sure it's necessarily so, but I suspect that it may be easier for a user to:

"Hit the button to download this thing that someone told me I should download and then I'll have a basemap whether or not I understand what just happened"

As opposed to:

"Download a specific MBtiles file from my browser, know where it went in my local storage (probably my Downloads folder which is also full of all kinds of crap) and then point ODK Collect to it."

But you're right, download files directly from a URL can be brittle.

How to communicate it to users

Copying from a message isn't great, but it's not that bad either. Long-press on a URL in a WhatsApp, Signal, or SMS message usually gives you the option to copy the URL in a way that pastes reasonably reliably.

For my purposes, I'd most likely make MBTiles files with one or another external utility (either a GIS for custom tiles or a [ethical] scraper that grabs tiles from a TMS and stuffs them into an MBtiles file) and stick them on a web server (for some people I imagine they'd be pretty happy if they could shove them up to a Google drive, drop box, or something like that and serve download URLs that reach them there).

Splitting subjects

Yeah, good call!

Renaming thread

By all means!

13

For the 'hit button to download after pasting a URL' method, if the files are somewhere that requires a user/pass/2FA then it may not work?

Stoked to see this discussion so active though, I find offline tilesets incredibly useful and anything that makes them more frictionless to deploy/update/specify can only be good.

2 Likes
14

Similarly stoked!

In the case an MBTiles file stored somewhere with some PITA constraint like 2FA, username/password, or whatever, my immediate instinct as a project manager would be download the damn thing myself and and put it up somewhere without that constraint, and give my enumerators a URL to it there.

15

To be sure I am understanding it well, from my farm mapping perspective :
Does this allow me to add on mbtile per fram within the same form, as any other media. And when a new farm is involved in the project, I'll just need to update the form by adding the missing mbtile ?

Also glad to continue this discussion :slight_smile:

16

@mathieubossaert that makes sense, with the caveat that MBTiles often differ from other media in one very important way: they're often enormous and impose a huge storage and bandwidth requirement on a server.

I'd be hesitant to set things up in such a way that the Central server is the only place you can store and serve them from.

If you can specify a URL to an arbitrary MBTiles file per form, you can—if necessary—store those big 'ol MBTIles files somewhere else, leaving your Central server less burdened.

And in some cases, the URL isn't actually pointing to a precut file, it's pointing to a machine that's got a bunch of geotif on it that's going to cut the MBTiles on the fly, which is in many cases much more efficient

17

I agree that I forgot this aspect :frowning: , and colleagues will suffer and blame me because of heavy form updates on their phone for mbtiles they don't need. It would be ok if I disable automatic form definition updates but it will create other issues I don't need !

My idea(l) would be something like this :

  1. My colleague open the form to map a farm
  2. A question ask him if he need a new basemap
  3. If yes, he selects the sites he wants to visit in a select widget. The form definition contains the url of each mbtile
  4. And then download the mbtile to use it as soon as the download is completed.

Or :

  1. My colleague opens the form
  2. A question ask him for the site he wants to visit
  3. Collect checks if the associated basemap (mentioned in the form definition) is available on the phone
    3.1 if not then download it
  4. Uses it

In that case I'll still just need to update the form, by adding a new site and its dedicated mbtile's URL in the form definition.
Updating forms on devices will be very light.

In the end any solution will be fine for us if we can let the users of a same single form to choose between listed mbtiles to download and use.

1 Like
18

If you use my Field Mapping Tasking Manager, you can:

  • Use QGIS to create a layer with one polygon around every farm.
  • Upload those polygons as GeoJSON to the FMTM. It automatically creates a project with one form per farm, and also cuts one MBTiles file for each farm from the TMS of your choice.
  • it also generates a QR code for each farm, which links to the appropriate form and any media associated with it per farm.
  • As soon as any of those QR codes are loaded into ODK, it immediately configures that project and downloads the relevant media. 1 farm -> 1 form -> 1 set of media files -> 1 link to a downloadable MBTiles file covering only that farm (and, of course, anyone can load up as many QR codes as they like to map multiple farms).

That's why I want the MBTiles downloader, and in the long run, would dearly love to be able to specify that file in the form! At the moment I'm forced to ask every mapper to use a USB cable to sideload the base map for each farm. But as soon as I can specify a file for each form, every mapper will literally have only the exact base map tiles needed for their exact task areas. They will be using no more bandwidth/storage than necessary for their task area(s).

If this sounds useful for your farm mapping, let me know, I'd be delighted to set up an FMTM task with you to see if it is useful for your team.

1 Like
19

Really interesting thread, and glad to see the proposed solutions to getting offline background maps onto a Collect device! There are some definite parallels with how the Digital Democracy team is approaching integration of mbtile background maps in Mapeo, with Downloader-esque functionality in the Desktop app and Picker functionality in the Mobile app: link and link.

I think that both Downloader and File picker concepts have respective merits and having the option of both seems to be ideal. In favor of having a File picker avbailable at the very least, I want to add the user story of retrieving map tiles onto devices that are mostly offline, or where internet is expensive. I have worked with some Indigenous communities in South America and East Africa that are quite handy in sharing files between devices over Bluetooth or using a tool like Xender, and in a recent mapping project, a community in Kenya actually shared mbtiles for use in Mapeo in this way. Having a File picker in Collect would enable a similar workflow where many devices can receive an mbtiles file without all of them needing to come into connectivity, or utilize their data plans to retrieve the same file.

One of the downsides of the File picker workflow is that it essentially requires duplication the file by copying it into the internal Android directory of the app. As already mentioned, mbtiles can be quite large in size, and deleting the original file in the device Downloads might be a step that a lot of users neglect to do. I'm curious if anyone has thought about this. Maybe there is a way for the UI to prompt the user to delete the original file and initiate an intent to open a file browser in the right directory where the downloaded mbtiles file is located, once the import has finished?

20

Though my team's needs would probably be somewhat better served by a downloader, it's true as @rudo says that people in low-connectivity environments are often quite adept at file-swapping, and therefore might be particularly happy with a picker.

*In my case, I usually want individuals doing distinct separate areas, so I mostly want to give everyone a unique MBTiles file; local file sharing won't be that useful because one person's basemap won't cover the other person's task area.

But I have a stupid question: do we have to duplicate the file by copying it into the internal Android directory of the app? Why can't we just read it from the Downloads directory directly? If we can copy it, that means we can read it; why copy it then? (don't get me wrong, I hate the idea of having lots of random MBTiles files in my Downloads directory, but I think I hate the idea of duplicating the same big file in several directories even more).