How should ODK Collect style vector maps?

One of the new things in Collect 1.23 is support for vector MBTiles in the geospatial activities (GeoPoint, GeoTrace, and GeoShape). Previously, you could put a raster MBTiles file in the /sdcard/odk/layers directory, and it would appear in the menu of available basemaps; selecting it would case the raster tiles to be shown on the map. In 1.23, you can also provide an MBTiles file that contains vector tiles. It will appear in the menu of available "Reference layers", and it will be drawn on top of your selected basemap.

However, vector MBTiles files don't include styling information, as far as I know. As a temporary measure, ODK Collect 1.23 draws the vector lines in various colours, choosing a random (but stable) colour for each layer. But the result is still somewhat strange-looking and unfamiliar, especially if you're used to seeing nicely rendered OSM maps.

(In Mapbox terms, we have a source of data, but we don't have any layers that describe how to render the data. The way we construct the map now is by starting with the basemap style (e.g. Mapbox Streets), then adding the MBTiles file as a source, and finally generating a simple line-drawing layer for each vector layer in the MBTiles file. We pick a colour for each layer by hashing the layer name.)

So now we have a design question: how should styling information be provided?

These are a few (non-exclusive) options we have considered so far. (In the below, I have mostly assumed that "style file" means a Mapbox JSON style file, though I'm open to other solutions.)

  1. Apply a default style file. We don't know if one style file will generally work for all or most of the MBTiles files people will use in practice. It seems like they would often be extracted from OSM data, so perhaps one could assume a standard set of attributes? Is it conceivably possible that a default style file would do the trick?

  2. Provide a list of stock style files. If there are a small number of common style files that would cover most cases, perhaps we can provide those as options. How realistic is this?

  3. Allow a style file to be sideloaded onto the phone. This would be the most flexible, and this functionality seems necessary for any serious use. But how should a style file refer to an MBTiles file on the local filesystem? In the Mapbox JSON format, tile sources are specified with URLs. (Actually, each source gets a list of URL templates, so that the load can be spread over multiple tile servers.) I don't see anything in the current spec about referencing local source files. One possible extension to the Mapbox JSON spec would be to allow the relative path to an MBTiles file as the "tiles" property of a source (it would be one string instead of an array of string); or to define a new property for this purpose.

  4. Include style information in the MBTiles file. Using a style file would be the most powerful method, but it would require managing multiple files, and creates the possibility of broken references if the source files have the wrong names or are in the wrong locations. As an alternative, we could define a way to embed styling information in the MBTiles file itself, so that each MBTiles file is a self-contained, viewable object. For example, we could add a "layers" property to the metadata table, containing a JSON array of layer definitions.

Finally, I should note that Mapbox has an existing method for achieving roughly similar goals; there is a way to define an offline region and download an offline package that contains all the tiles and styling resources. However, I don't believe the format for this package file is public.

We're looking for suggestions and guidance from you—our Mapbox friends, developers, and implementors who dream of being able to use ODK Collect fully offline. For the use cases you have in mind, whether it's using offline tiles to serve as a basemap, replacing the existing online basemaps; or whether it's using offline tiles as a reference overlay—what is workable for you? Can you describe the process you imagine using to deploy a collection project with offline maps in the field?

As always, thank you for your wisdom and input!

1 Like

Tagging a few folks who I'm imagining will have something to say about this: @Marena @langstonsmith @mikel @Ivangayton @paul.uithol @smit1678 @LN @mathieubossaert

Please feel free to invite others!

Passing this along from Android devs at Mapbox:

I should note that Mapbox has an existing method for achieving roughly similar goals; there is a way to define an offline region and download an offline package 1 that contains all the tiles and styling resources. However, I don't believe the format for this package file is public.

This probably would be the easiest path for us to support and for ODK to be able to have more natural controls over the map style. Mapbox's offline functionality downloads all the necessary resources for a style and it's compatible with our regular source/layer features. Finally, note that this package schema for offline is public at https://github.com/mapbox/mapbox-gl-native/blob/master/platform/default/include/mbgl/storage/offline_schema.sql . (However, it's worth noting that despite being SQLite-based like MBTiles it is a different format than MBTiles.)

Curious if there's interest in reviving this topic. There are so many interesting questions and ideas here - I, for one, would find it helpful to understand more about applied use cases involved and maybe define a more specific, common pain point.

  • Should we focus primarily on offline use-cases here?
  • Is the priority need to support adding custom MBTiles (e.g. building footprints from OSM) as layers on top of a standard basemap? Or is it about custom MBTiles that are acting as the entire basemap?
  • Or is the need more about how to cache and access a standard styled vector basemap (e.g. Mapbox Streets) while offline?
1 Like

Hi @Marena,
I was just returning to the forum to see if there had been any change to the vector maps issue and saw your post.

Most of my work is done in areas with poor or non-existent data connections so offline use is what interests me...

I would like to be able to plot points and lines (a grid and a transect) as an overlay to guide enumerators to pre-defined locations. If for example I was able to style a line to have a marker at each vertex / node then this would be really useful.

Another example for my point layer is to be able to style it/them based on a 'property' of each point (assuming that it is possible to 'embed' that within the vector layer) - which would give different symbols or colours (or both). This would indicate, for example, that the point requires certain parts of the form to be filled in.

In other situations I suspect that if I could translate a series of vector layers with styling (Ordnance Survey Zoomstack, for example) this would be a more efficient storage method than converting to raster and loading as a layer in ODK. This would allow me to prepare the backdrop map on the desktop and transfer to a device, knowing that all enumerators have the data available.

To explain how I 'get round' things at the moment I produce a .mbtiles file in QGIS that creates raster tiles based on a defines area - I have full control over how that map appears through styling in QGIS so I have the backdrop mapping and my data layers with symbols and sometimes labels, but obviously this creates large file sizes if I have zoom levels that are useful for navigation to my predefined locations... Just using a single vector layer for my grid or transect would simplify things and make it easier to share assuming we can simplify how people can access these layers - see: Button to select an offline map file on the phone and move it at the right place - #2 by seewhy

Hope this helps explain potential use cases relevant to me... Of course, then I would need to learn / find a tool to create vector mbtiles...

1 Like

Thanks @seewhy to remind us this promising discussion and @Marena 's question.
I should have wrote the same answer (except Ordnance survey reference !)

For us, offline maps are more important as we work in not well wired zone.

Would be great to be able to add an overlay with only vector data, we should style from an attribute. For example, to show what material need to be repaired, which parcel of land we own (police mission)...

As @seewhy said this can already be easily done with QGIS.

The tool side of 'how to style vector tiles" is important as for the moment I do not know such a tool. QGIS will have more and more support for vector tile but for the moment it is a bit "uneasy"

Just came across this thread as I'm working my way through using offline maps for the first time in production.

My use case is of course related to turtles.

  • A dozen separate data capture programs spread across 2000 km of coastline, all capturing data while offline
  • Our offline raster map of turtle nesting beaches is fairly static (unless cyclones re-arrange our coastline) and can be distributed onto each data collection device once per season (ca 4GB all up)
  • Potential vector overlays are small but could be updated between once per season (monitoring sites) and daily (e.g. some marked turtle nests requiring follow-up)

Our current setup works with all data (raster and vector) distributed as raster basemap.
An improvement were if we could split the huge and static rasters from the tiny and often changing vecors:

ODK Collect could support an offline raster basemap (as it does now) plus at least one additional vector overlay with styling.

As for the vector styling, I could imagine the demand for a robust default (as it is now?), some built-in styles, plus support for whichever styling standard is easiest for ODK users to generate.
In our raster basemap I'm including a vector layer showing our monitoring sites (turtle nesting beaches) as 25% opaque polygons together with the site names as black text with white border and drop shadow (for readability).

ODK Central could push out updated vector overlays to separate app users or to all app users (like form media).
We have one app user per monitoring program. We use the same forms across all programs but have no exchange between them (they are hundreds of km apart and consist of separate teams).

1 Like

I would really like to help with this. I do have some development skills though never contributed to project like this. My pick would be definitely:

    1. Include style information in the MBTiles file

As @Marena pointed out the Mapbox specification for doing exactly that (the file has by default extension *.db) is publicly available at https://github.com/mapbox/mapbox-gl-native/blob/master/platform/default/include/mbgl/storage/offline_schema.sql

I had spent some time studying the schema, and converting between db and mbtiles standard, to make sure it is possible to create db package with open source tools.

Now if someone experience with development of ODK could point me out where to start:

  1. Mbtiles file is saved in layers folder to make it available in the app, the same would apply to db
  2. The tiles ar served from mbtiles file via localhost server? That is my understanding from reading some post from 2019.
  3. Would the right solution be to extend serving 'files' by extending what is served by localhost from db file? - Style file itself, sprites json / png, fonts pbf?

Thank you in advance for any guidance on this.

2 Likes

Thanks so much for picking this back up, @mirouml. This is a problem we'd really like to solve for users and that we absolutely need feedback and collaboration on.

A few things to consider:

  • Some users like the ones in this thread want to use custom basemaps or layers. Many want a generic offline basemap for the region they work in. Some want both.
  • Whatever we come up with has to be relatively easy to create artifacts for using open tools (as you focus on).
  • We likely won't be able to rely on directly pushing to the device for much longer (Android 13 removes most options other than adb: https://github.com/getodk/collect/issues/5304).

I'm very interested in learning more about what you found and how you ended up producing these files. As you note, that is a big consideration when selecting a format.

Yes, that's what I would expect.

4 posts were split to a new topic: Future of device access to copy custom basemaps

Thank you.

Some users like the ones in this thread want to use custom basemaps or layers. Many want a generic offline basemap for the region they work in. Some want both.

Yes. The idea is certainly to support both. There is an option out there to get OSM vector offline mbtiles - free for non-commercial projects. AND BSD licensed styles and fonts for these standard OSM tile packages. For non-technical people probably the best way to get all needed to use offline basemap. Sure some straightforward tutorial needs to be written for that, and ideally the font and style files being available by ODK. For custom creation there is maputnik.

Whatever we come up with has to be relatively easy to create artifacts for using open tools (as you focus on).

That is a very good point. There is an open source SDK MapLibre, which took up from where open source MapBox SDK finished - and actually allows using vector mbtiles file + external fonts as pbf files, style as json etc. directly from local folder. To convert between this and standard Mapbox db format I wrote a quick and dirty python script for now. Anyway developing and using extra tool for conversion adds complexity, so maybe using MapLibre approach might win this?

We likely won't be able to rely on directly pushing to the device for much longer (Android 13 removes most options other than adb: https://github.com/getodk/collect/issues/5304 3).

My focus here is really to get the styles going. Maybe this can help with that issue - https://www.reddit.com/r/Android/comments/wsnhct/file_manager_dev_discovers_a_new_loophole_to_get/
Or maybe start using Documents / Download folder for storing offline files would make it easier?