Adding Mapbox vector tile basemaps

Ideas
1

Hello everyone! I work at Mapbox on our Community team, which supports non-profits, researchers, and other positive impact projects to use our tools. ODK Collect is a very common data collection tool among our partners. Earlier this year I connected with @yanokwa about the possibility of integrating Mapbox basemaps (which are built largely with OpenStreetMap data) as an option within ODK. A colleague of mine on the Mapbox mobile team has been volunteering his time to investigate what would be involved to do this

I wanted to start a topic to let you know what we've been trying and invite discussion and feedback about whether and how to add this feature.

What is the general goal of the feature?
While OSM Droid (currently in ODK) does support Mapbox, it only supports raster tiles not vector tiles, which are faster and lighter to use and are what most mobile maps are moving to. We think vector tiles would also help with low-bandwidth and offline mapping scenarios.

Rather than sinking time into a Mapbox raster option (which might be more complicated and not as good), it looks like the easiest thing to try would be adding the Mapbox Android SDK to ODK, alongside the other map options. That would allow for vector maps and would benefit from our optimized offline caching. And it would mean there are more folks at Mapbox who would be able to help!

Another benefit of using the Android SDK is that each user will have their own ceiling of 6,000 tiles for offline caching (the ODK app will need to manage the tile cache). Users would not need their own Mapbox accounts, you could use one token from an ODK Mapbox account.

What are some example use cases for this feature?
The main driver for this interest is for a better mapping / map view experience when collecting data offline or where there is low connectivity.

The other driver would be to give ODK users the option of using Mapbox basemap styles in their map interface - and potentially even letting users use their own custom Mapbox styles if they've made one.

What can you contribute to making this feature a reality?
From Mapbox's side, we can contribute technical and accounts support.

  • From the technical implementation side, my colleague Langston Smith (and engineer on our mobile team) is looking at the current ODK implementation to see what it would take to add in Mapbox vector tiles, and potentially get from raster to vector tiles across the board. He's volunteering his time on this so is working through it bit by bit. So far he's got it so Mapbox shows up in the options:
    odk%20post

  • From the accounts side, I can help work through questions about what user token would be used. To simplify things, and keep with the ODK approach of not wanting to require users to get their own map provider tokens, it might be easiest to create a master ODK Mapbox account and have that account's token built into the app - that way we could work with ODK to monitor that traffic and cover it for free. But we need to look at estimates for tile use and caching activity to get a sense of how much traffic we'd be looking at.

    • It might be a good idea to require (or request?) users to set up their own Mapbox accounts if they are planning a fairly large deployment, or if they plan to go above a certain level of usage. (And the Mapbox Community team could help those users if costs are a concern.)
    • An incentive for users to create their own Mapbox account could be that the app could maybe allow them to use their own custom style basemaps. We could also help create some documentation about working with data from ODK in Mapbox tools, to take advantage of other aspects of having a Mapbox account.
    • If you'd be interested in using your own Mapbox account (or at least be willing to), that would be great to know!

  • If we can figure out the basemaps, there might be other ways that Mapbox volunteers can contribute to helping to strengthen the geo tools within ODK.

What do you think? Would you like to see Mapbox vector tiles as a map option in ODK Collect? Are there any concerns or questions we should have on our radar as we try this out?

2

Interesting. Mapbox is great technology however licensing is an issue and it seems to me the license has been getting progressively more restrictive recently (just an impression I may be wrong.

You say "it might be easiest to create a master ODK Mapbox account and have that account's token built into the app - that way we could work with ODK to monitor that traffic and cover it for free". Would you also provide that free service for odkCollect forks?

3

That's a great point about forks @Neil_Penman - figuring out the tokens is one of the trickier questions, but we want to work with everyone to figure out a good solution. I suspect that for forks we'd want to ask that folks get in touch with the Mapbox Community team to set up their own token so that it doesn't all go through the ODK account. That way we can understand the fork's specific use case and if their traffic might surpass our free threshold we can set them up with their own coupon if they are a non-profit / positive impact / research use case. (If it is a commercial use case we couldn't guarantee a coupon.)

4

@Marena Pardon my ignorance here, but do you know if the Mapbox vector tiles would allow for non-Mapbox vector tiles?

5

@yanokwa Langston is looking at how to implement Mapbox vector tiles specifically, via our Android SDK, but I think his experience could inform what would be required to add in other vector tiles as well. Has anyone else in the ODK community been working on vector tile options?

1 Like
6

Hey folks. As @Marena said, I've been slowly adding Mapbox to the ODK Android app. Here's where I stand

I'd love for question/discussions about Mapbox/ODK collaboration to continue!

7

@langstonsmith A very welcome to the ODK community and thank you for actually building something we can play with! Is the code somewhere that we could take a quick look at? I'm particularly curious how big of a change this is (both in code and the resulting binary).

@Neil_Penman Was @Marena's answer to the token issue acceptable? It feels like a similar setup with Google Maps where ODK as a project has an API key that supports everyone who uses the app and forks can use their own key. Agreed that Mapbox licensing is getting restrictive, but so is Google's. We've tried to hedge a few years back by bundling OSMDroid which is unrestricted.

8

Hi @yanokwa

My work is slowly progressing at https://github.com/opendatakit/collect/compare/master...langsmith:ls-adding-mapbox-maps . I needed to added Mapbox activities for the 3 types of geo-based widgets. Also need to integrate device location (https://www.mapbox.com/android-docs/plugins/overview/location-layer/ ) into the button functionality in those activities.

I've got lingering concerns about whether the Mapbox Maps SDK for Android can support the app's offline requirements, but I guess I'll cross the bridge if/when I come to it.

1 Like
9

@yanokwa, are there shareable stats on the Google Maps API key usage that might give an initial idea of the load that might occur on a Mapbox token?

Would this change replace/eliminate the ability to use raster tiles? Does the change let you use a mbtiles file of vector tiles and how does this affect using offline maps? @Marena, you mention:

adding the Mapbox Android SDK to ODK, alongside the other map options

but I'm slightly confused by:

and potentially get from raster to vector tiles across the board

10

@danbjoseph we use ~150K requests a month for Collect which is pretty low and fits well below the $200 monthly credit that Google provides to project.

11

Tom from the TSC here. I don't see any reason not to permit this from a licensing standpoint, and I think it would be a nice addition.

I wonder about offline support. How well do the other maps work offline? What are our standards in this regard?

1 Like
12

@tomsmyth There's a good write up on offline maps support at https://docs.opendatakit.org/collect-offline-maps. Both existing SDKs support loading tiles from the SD card.

13

Sorry for the confusing comment @danbjoseph- I didn't mean to imply a forced change off all raster tiles or a removal of raster functionality - rather, we're looking at an addition of vector tile options. If we can figure it out for Mapbox vector tiles, then maybe that would open the door to other vector tile possibilities more generally (@yanokwa wondered whether this effort might be able to help advance on that).

The hope is that if we can get this working, the Mapbox vector tiles would provide a good option for offline maps, but that's one trickier piece that @langstonsmith is looking into.

14

I have to be upfront here and say that I don't know a ton about mapping technologies so I may have some rather elementary questions! Thanks so much for taking the initiative on this and I'm sure we will end up with a great feature for users!

When you're talking about Mapbox vector tiles, what exactly does that mean? From what I can see, it sounds like it would be mbtiles with pbf format tiles, is that right? I'm having trouble determining exactly what pbf is. I see https://wiki.openstreetmap.org/wiki/PBF_Format which appears to define the standard and provide a reference implementation but at https://github.com/mapbox/mbtiles-spec/blob/master/1.3/spec.md#content, pbf is defined as "gzip-compressed vector tile data in Mapbox Vector Tile format". Are those the same things or are there two non-compatible standards for pbf?

Does the Mapbox Android SDK only support tiles from Mapbox or could it also provide support for tiles from somewhere else? @langstonsmith have you also looked at supporting vector tiles in osmdroid (e.g. https://github.com/osmdroid/osmdroid/issues/101) and if so, what might be the tradeoffs there?

I was involved in a quick discussion with some folks from Humanitarian OpenStreetmap and they were enthusiastic about support for vector tiles. They also brought up that GeoPackage is becoming more and more common and has more flexibility than mbtiles. Does Mapbox provide assets in that format or offer any support for it?

I see two different aspects to offline maps: caching and direct transfer. That is, a user might load a map from their device and then want the tiles that they visit to be accessible for later offline use. Alternately, a user may want to transfer over a tile set without ever needing to talk to a server (https://docs.opendatakit.org/collect-offline-maps/) as @yanokwa mentioned above. @Marena addressed the caching piece in her initial post so am I correct in assuming it's the general transfer of tiles that's less clear? I think this ties in to my previous question about alternate tile sources.

There's some work that has started in the mean time to consolidate the two currently-available mapping engines and the different geo activities. @zestyping is starting by providing a common interface for mapping implementations (currently GMaps and OSM; adding Mapbox down the line). A first draft is available at https://github.com/opendatakit/collect/pull/2465. Next steps will include factoring out common code between the various geo activities.

This poses a bit of a sequencing challenge. Perhaps it makes sense to focus on a reference implementation for one of the geo activities? Then hopefully it should be relatively straightforward to generalize once the new architecture is in place. @zestyping, what do you think and do you have any more questions for @Marena and @langstonsmith that may not have come up yet?

15

Thanks @LN - there's lots of pieces in there, most of which @langstonsmith is probably better able to answer than me. I tend to refer to our documentation like https://www.mapbox.com/help/mobile-offline/ and https://www.mapbox.com/android-docs/maps/overview/offline/ but these questions seem deeper than that, so I'll defer to @langstonsmith.

For the question about GeoPackage, I'm not familiar with that system - I'll see if I can learn more from HOT and OGC folks at FOSS4G later this month.

And for the question about caching vs direct transfer. Mapbox Terms of Service only allow for caching directly on a device for offline use, not direct transfer or 'side-loading'. We have created some special permissions for certain use cases, like offline community mapping, but we need each organization to get in touch with us for that special permission. So for the general Mapbox tiles in the ODK Collect app, it would have to rely on caching - if particular projects or organizations needed to use direct transfer they could connect with us.

1 Like
16

Hello, @Marena. I'm going to be at FOSS4G as well—looking forward to meeting you there! Thanks for offering this contribution and helping us to figure out how the use cases will work.

As things stand in ODK Collect in the moment, for each type of geographic data widget, there are two entirely separate activities, one that uses Google Maps and one that uses OSM. So we've got two GeoPoint activities, two GeoTrace activities, and two GeoShape activities. This means lots of code duplication and inconsistencies in behaviour between the two implementations in each pair. It also means that adding support for another map SDK (such as MapBox) involves writing a third activity of each type, duplicating even more logic.

The work I'm doing right now is trying to improve the situation by defining a common interface across all the map SDKs. Here's that interface definition:

I'm writing two implementations of this interface, one that uses Google Maps and one that uses OSM. The plan is that we'll only need one activity of each type (GeoPoint, GeoTrace, and GeoShape), and the nothing in the activity will be specific to the map SDK.

@langstonsmith, thanks for the work you're doing! The functionality looks great in that animated GIF you posted. I realize that the new MapFragment interface will mean that some significant refactoring would be required in order to integrate your MapBox work, and I'm sorry to be the bringer of bad news. But, we do want and intend to support many options for map SDKs, and I hope it makes sense why we're moving in this direction!

—Ping

17

No worries @zestyping. Thanks for the update.

It also means that adding support for another map SDK (such as MapBox) involves writing a third activity of each type, duplicating even more logic.

Was almost finished too :wink: No problem at all. I'll hold off on further work until your interface is finished and I can refactor accordingly.

18

@langstonsmith Yeah, I'm sorry—I feel bad about making you do more work!

We've reached the point where a first draft of the common interface is ready, and has implementations for both Google and OSM that work for the GeoShape activity (see PR 2465). At this point I expect it to be fairly stable, except for the addition of methods to show editable point markers (for GeoPoint) and polylines (for GeoTrace), which will follow the pattern of the existing methods for showing editable polygons.

I'd welcome your feedback on it—would you mind having a look at the MapFragment interface and letting me know if you see anything about its design that would make it difficult to implement with MapBox?

1 Like
19

I feel bad about making you do more work!

No worries. I learned a ton about ODK and the app! We're making progress :slightly_smiling_face:

I probably won't have time to truly dive into the #2465 pr (and maybe add a Mapbox map fragment) until the 2nd half of next week :unamused: This refactoring will be a good test of the Mapbox Maps SDK's SupportMapFragment !

We recently added draggable markers, so I need to figure out how that might fit into ODK.

I think I'm going to keep waiting to see what you end up with and then see how Mapbox fits into it all

20

Hi all, Paul from Humanitarian OpenStreetMap team (HOT) here. Access to and interfaces for using vector tiles sounds like a great feature to have! For accessibility and ease of use, having a default Mapbox tiles option would be a great place to start.

I think that ideally, integrating vector tiles would contribute to, and work towards more effective geospatial data collection in ODK across the board, and not be a parallel effort limited to use as a 'plain' basemap. I fully understand there are tradeoffs involved on Mapbox's end, but some of the complications/limitations I see are that Mapbox tiles limit the features they contain and expose, it'll probably be complicated to access to underlying OSM id's (and might lack the ability to attach extra data/attributes from form questions), and that portablity is potentially limited.

Some questions/suggestions from my end (these do mirror @LN, @yanokwa, and @danbjoseph's to some extent):

  • Would it be feasible to define a "map view" that can use configurable vector and raster sources? And could we define a vector tile interface, for which Mapbox is one of the (default) providers? For example, for cases where the Mapbox tiles don't suffice, we could then also have the option to package vector tile data for deployments, or set up vector tile endpoints containing a specific AOI and data types we need (say, all WaSH facilities + attributes we need for those).
  • As @Marena indicated, it would be awesome if the vector tiles would be stylable via MapCSS, or another (portable/standardized) format!
  • As @danbjoseph requested, designing to allow offline usage would be a major benefit. ODK is used by many humanitarian organizations in low (or non-existant) bandwidth environments. Caching is one thing, but requires those specific devices to be connected at some point. In a lot of our projects we need to be able to package and preload all map data in advance (aerial and raster basemaps as mbtiles, editable map data as .osm) before embarking.
  • What options do you see if we were not just to think about displaying basemaps, but also enhancing editing and geospatial data collection? Would that somehow be feasible on vector tiles? What (standard) file formats would be the most suitable for that? OpenMapKit has been using .osm xml, but I'd like to see us move towards open (OGC) formats if possible, like GeoPackage. Thoughts?
1 Like