Generate offline background imagery for ODK Collect (MBTiles)

Generate offline background imagery for ODK

Following on form the ODK Collect docs on offline maps, this write-up is a brief worked example of step 1 of the quickstart.

The input are freely available* raster and vector data sources plus own reference data.
(*Reminder: Adhere to the copyright and usage terms of the imagery used.)

The output is one or several MapBox Tiles (.mbtiles) files placed into a mobile device's ODK Collect layer folder at /Android/data/org.odk.collect.android/files/layers.
Note on the ODK Collect storage location: latest ODK Collect versions are forced by a change in Android policies to store app data in the internal storage. The SD card is no longer an option. Some ODK Collect installations leave some files behind on the SD card.
TL;DR: Upgrade to latest ODK Collect, let it migrate its storage, and use the file paths from this tutorial.

Software

Quantum GIS v3.14 or higher (download from here)

Load reference imagery and data

In this step, we will prepare the imagery in QGIS exactly like we'd like to use if offline in ODK Collect. We will:

  • Load some vector imagery as fallback, e.g. OpenStreetMap.
  • Load some raster imagery of sufficient resolution.

Lowermost layer: vector tiles

Middle layer: raster imagery

  • Layer > Add Layer > Add ArcGIS Map Service Layer
  • New
  • URL https://server.arcgisonline.com/arcgis/rest/services/World_Imagery/MapServer
  • Name as you like, e.g. ESRI World Imagery
  • Order layer above OpenStreetMap
  • Lowest visible scale of ESRI World Imagery is ca 1:1050 in populated areas and around 1:1150 in less populated areas.
  • Layer properties > Rendering > Maximum (inclusive) > 1:??? (lowest visible scale in your area of interest)
  • Setting the maximum rendering scale ensures that this layer will be hidden when zooming in further, revealing the OpenStreetMap layer below, instead of covering it with grey "No imagery available at this zoom level" tiles.

Topmost layers: reference vectors

  • Add any vector layers of your own with appropriate styling.
  • Consider layer transparency, labels with drop shadow, buffer, and scale dependent visibility.
  • The attached example project contains a GeoJSON layer containing one example location and styling.

Generate MBTiles

In this step, we'll generate the MBTiles file from the raster and vector layers shown in QGIS for individual locations we want to take offline.

QGIS can generate MBTiles natively, see the QGIS docs on Generate XYZ Tiles.

  • Open Processing toolbox > Raster tools > Generate XYZ tiles (MBTiles)
  • Zoom QGIS to each locality of interest and adjust the the raster layer's rendering maximum scale to the available scale. This step is necessary to avoid grey placeholder tiles overlaying the OpenStreetMap imagery.
  • Extent: draw on canvas
  • Zoom min 5, max 20 or to taste. Every additional "high" zoom level increases the resulting .mbtiles file size exponentially.
  • Tile format: PNG is better quality, but ca 20 times the size of JPG at 75% quality
  • Save to an output file (.mbtiles)
  • Review the resulting .mbtiles by opening in QGIS (drag and drop onto layers pane). Do all zoom levels work? Is the resolution sufficient? Iterate to taste. Save "good" settings as a batch job as shown in the attached example.

The MBTiles show the imagery and vector overlays at any zoom level just as they appear in QGIS.
Including reference layers, such as "my study sites" or "my known locations" is useful as context for data capture using the manual map widgets, and to review saved submissions in situ.

Transfer and test

Transfer the .mbtiles to your devices as per docs - copy into /Android/data/org.odk.collect.android/files/layers. Test and review raster zoom level, imagery resolution and extent. Especially getting the cut-over between raster and vector basemaps, set by the raster layer's scale-dependent rendering threshold requires some trial and error.

In forms, the maps work nicely with "manual (No GPS)" maps (appearance="placement-map") where users can capture a location other than their current one.
The other useful application is a background layer in the "map view" of saved forms to review already captured forms over the offline map.

Further considerations

Optimise imagery extent

There are tools available to further reduce .mbtiles size. An interesting approach is to exclude unnecessary tiles; e.g. if the area of interest is long, narrow, and diagonal (turtle nesting beaches come to mind), the square .mbtiles will contain two triangles full of images you'll never need.

Storage limits

Some OS limit the size of files you can easily transfer. Mobile data capture devices have limited storage. Our oldest devices (Lenovo Tab3) have 16GB all up, with ca 10GB available.

Running start

The attached example project contains:

  • A QGIS project file (.qgz) containing:
  • ESRI World Imagery as raster layer.
  • OpenStreetmap as fallback vector layer.
  • An example vector layer (areas.geojson) with one polygon.
  • A QGIS style file (.qml) for the example vector layer.
  • A saved batch job (.json) of the "Generate XYZ tiles" tool.
  • The resulting .mbtiles file from the batch job.
    Note: The selected area for the MBTiles export is tiny to keep the file size portable.
    ODK tutorial.zip (1.8 MB)
5 Likes

Hello, for some reason my layer files can't be located. I've followed your process but when I try to reference the layers it comes up with the message
There are no layer files in /...................

Can you help?

Regards

Hi @dominicgusah,
Welcome to the forum! If you have a moment, introduce yourself in the welcome thread!

What ODK Collect version are you running? The tutorial assumes the latest version, which changed the internal file path.

What file path are your .mbtiles files under on the tablet?

Can you open the .mbtiles in QGIS?

Thank you, Will do.

I'm using v1.28.4

I didn't realise there was a similar 'Layer' folder in Internal storage. I was using the one in the SDcard. When I saved the files to the internal folder it worked. This is despite the Reference setting saying "SDcard/......."
How do I now change Collect to look for the MBTiles in the SDcard and not on the internal storage? (As there is usually more space there)

Regards

Ah, understood, thanks for the clarification!
The issue you're experiencing is that the latest ODK Collect versions are forced by Android to migrate their storage to the internal storage. The SD card is no longer an option for app data unfortunately, which to my knowledge is an Android limitation.
So, the paths in my tutorial are correct for your version (1.28) and we should update the ODK docs https://docs.getodk.org/collect-adb/#collect-directory.

1 Like

Here's a quick preview of the example: (ping @LN)

The study area my enumerators want to select is shown as a transparent, light blue polygon with a 1px border over the aerial imagery. The study area is labelled with its name, which means something to my enumerators (here: Cape Domett in the Kimberley region of Western Australia) and the primary key (145) from the database it's from (purely for my own convenience).

2 Likes

Some notes after further testing:

  • Use 0 as the minimum zoom level, not 5 as I stated above. The top-most zoom levels cost next to no storage space and generate really quickly. Using 0 also means that the imagery is always visible when zooming and panning at extreme values.
  • Use 19 as max zoom level. The default zoom of ODK Collect's map seems to be 19 (cf. https://github.com/getodk/collect/blob/master/collect_app/src/main/java/org/odk/collect/android/geo/MapProvider.java#L91), so if your offline map renders nicely at that level, the storage and processing cost of adding level 20 is up to you.
  • The QGIS plugin "Zoom Level" is useful to calculate the zoom level at a given scale. The relationship between those depends on latitude.
  • The MBTiles do not exactly follow the render cut-off scale. E.g., even if I configure the maximum zoom ESRI World Imagery's scale dependent visibility to go to a maximum of 1:1100 (this varies between 1000 and 1200 across Western Australia, YMMV), it should cut out before I see the grey placeholders saying "Map data not yet available" hiding the OpenStreetMap vectors. However, I can see the grey placeholder tiles for the entire zoom level 19 (roughly a scale of 1:1100 down to 1:500 at my latitudes of -20 degrees).
  • To get a nice cut-over of raster aerial imagery to vector basemaps you might need to find a scale dependent visibility maximum for the raster that's just at zoom level X.001 or Y.999 or so. Let us know here if there's a formula.
  • Getting the offline maps just right requires a lot of testing and verifying. Make sure you send your folks out with sufficient imagery in extent and zoom level.
1 Like

@Florian_May I just wanted to thank you for this guide, it was helpful in getting me started in creating my own MBTiles as someone who has never done any GIS before.

I have now georeferenced existing layout drawings, and combined them with map and satellite layers and scale dependent visibility, it's fantastic. :world_map:

2 Likes

Hey thanks for the feedback @ahblake, that's awesome to hear!

A colleague question reminds me that I forgot to share a fairly faithful french translation of Florian's showcase :frowning: , made with its agreement :wink: one year ago and adapted to French context (data providers)
Here it is :

####################

Voici la traduction d'un tutoriel de Florian Mayer sur le forum ODK, qui présente comment générer vos fond de cartes avec toute l'info que vous voudrez y superposer, pour aller sur le terrain avec ODK ou Oruxmaps (navigation seule)

Article Original : Generate offline background imagery for ODK Collect (MBTiles) - #6 by Florian_May

Générer des fonds de carte hors ligne pour ODK

La documentation sur l'utilisation de fonds de carte hors lohgne dans ODK est ici : (https://docs.getodk.org/collect-offline-maps/), cet article est une mise en partique rapide de l'étape n°1 du [quickstart] (https://docs.getodk.org/collect-offline-maps/#offline-maps-quick-start).

Les données de base sont des sources de données raster et vectorielles librement disponibles*, ainsi que des données de référence propres. (*Rappel : respectez les droits d'auteur et les conditions d'utilisation des sources de données utilisées).

Le processus aboutira à la création d'un ou plusieurs fichiers MapBox Tiles (.mbtiles) placés dans le dossier dédié d'ODK Collect /Android/data/org.odk.collect.android/files/layers. Remarque sur l'emplacement du stockage d'ODK Collect : les dernières versions d'ODK Collect sont contraintes par un changement de la politique d'Android vis-à-vis du stockage de données de l'application dans le stockage interne du téléphone. La carte SD n'est plus une option. TL;DR : Passez à la dernière version d'ODK Collect, laissez-le migrer son stockage et utilisez les chemins de fichiers présentés dans ce tutoriel.

Logiciel

Quantum GIS v3.14 ou supérieur (à télécharger ici)

Charger les fonds de cartes et les données de référence

Au cours de cette étape, nous allons créer dans QGIS, tlle que nous aimerions l'utiliser sur le terrain dans ODK Collect. Nous allons donc :

  • Charger un fond de carte "de secours" comme OpenStreetMap qui s'affichera si les couches supérieures ne sont pas dispon,ibles au niveau de zoom demandé.
  • Charger des images aériennes d'une résolution suffisante (les ortho-images de l'IGN)

Fond de carte : tuiles OpenStreetMap

Couche intermédiaire : photos aériennes

  • Couche > Ajouter une couche > Ajouter une couche WMS/WMTS
  • Nouveau
  • URL https://wxs.ign.fr/choisirgeoportail/geoportail/wmts?SERVICE=WMTS&REQUEST=GetCapabilities
  • Nommez-la comme vous le souhaitez, par exemple IGN ORTHOPHOTOS.
  • PLacez la couche au dessu d'OSM
  • L'échelle visible la plus basse de l'Orthophoto est d'environ 1:850
  • Propriétés de la couche > Rendu > Cocher "Visibilité dépendante de l'échelle" > Maximum (inclusif) > 1:850 (échelle la plus basse visible dans votre zone d'intérêt)
  • La définition de l'échelle de rendu maximale garantit que cette couche sera masquée lors d'un zoom plus important, révélant la couche OpenStreetMap située en dessous, au lieu de la recouvrir de tuiles grises "Aucune imagerie disponible à ce niveau de zoom" (ne se présente pas avec l'IGN)

Couches supérieures : données vecteurs de référence

  • Ajoutez vos propres couches vectorielles avec un style approprié.
  • Pensez à la transparence des couches, aux étiquettes avec ombre portée, aux tampons et à la visibilité en fonction de l'échelle.
  • L'exemple de projet ci-joint contient une couche GeoJSON contenant un exemple de localisation et de style.

Génération des MBTiles

Dans cette étape, nous allons générer le fichier MBTiles à partir des couches raster et vectorielles affichées dans QGIS pour les zones que nous souhaitons embarquer dans ODK pour un travail hors connexion.

QGIS peut générer nativement des MBTiles, voir la documentation QGIS sur [Generate XYZ Tiles 3] (https://docs.qgis.org/3.16/en/docs/user_manual/processing_algs/qgis/rastertools.html#generate-xyz-tiles-mbtiles).

  • Ouvrez la boîte à outils de "Traitement" > Boite à Outils > Outils rasters > Générer des tuiles XYZ (MBTiles).
  • Zoomer le canevas sur chaque zone d'intérêt.
  • Extent : Vous pouvez soit choisir l'étendue actuelle de la carte, soit l'étendue d'une couche, soit dessiner la zone que vous voulez générer (Dessiner sur le canevas)
  • Zoom min 5, max 20 ou selon votre goût (18 pour l'IGN). Chaque niveau de zoom "élevé" supplémentaire augmente la taille du fichier .mbtiles produit de manière exponentielle.
  • Format des tuiles : PNG est de meilleure qualité, mais environ 20 fois plus lourd que le JPG à 75% de qualité.
  • Enregistrez dans un fichier de sortie (.mbtiles)
  • Testez les .mbtiles résultants en les ouvrant dans QGIS (glisser-déposer sur le panneau des couches). Tous les niveaux de zoom fonctionnent-ils ? La résolution est-elle suffisante ? Faites des essais selon vos goûts. Enregistrez les "bons" paramètres sous forme de travail par lot 'petite disquette dans la berre d'outils.

Le MBTiles affiche les fonds de cartes et les données à n'importe quel niveau de zoom, exactement comme elles apparaissent dans QGIS. L'inclusion de couches de référence, telles que "mes sites d'étude" ou "mes emplacements connus" est utile comme contexte pour la saisie de données.

Transfert et test

Transférez les .mbtiles sur vos appareils comme indiqué dans la documentation

  • copiez-les dans /Android/data/org.odk.collect.android/files/layers. Testez et examinez les niveaux de zoom, la résolution dedu fonds de carte et l'étendue.

Autres considérations

Optimiser l'étendue de l'imagerie

Il existe des outils permettant de réduire davantage la taille des .mbtiles. Une approche intéressante consiste à exclure les tuiles inutiles ; par exemple, si la zone d'intérêt est longue, étroite et diagonale (on pense aux plages de nidification des tortues), le carré .mbtiles contiendra deux triangles remplis d'images dont vous n'aurez jamais besoin.

Limites de stockage

Certains systèmes d'exploitation limitent la taille des fichiers que vous pouvez facilement transférer. Les téléphones ont un stockage limité.

3 Likes

Bonjour,

je vous partage mon expérience pour afficher une couche de travail sur le module carto d'ODKcollect,
mon but étant d'afficher "une couche" de parcelle étiqueté stocké localement par dessus la cartographie satellite en ligne.

I share my experience with you to display a working layer on the ODKcollect carto module,
my goal is to display "a layer" locally stored of labeled parcel on top of online satellite mapping.

TutoMbtiles.pdf (1.0 MB)

Exemple de résultat :


2 Likes

If you have an opensource solution to merge two mbtiles corresponding to areas that do not overlap I haven't found the solution. When i have two layers with an high zoom, I have to load the 2 layers on the smartphone, and the final user must switch from one to the other.

I'm now using mapbox as a baselayer (has two closer zoom levels than GMaps, one closer than OSM) and now want to utilise the pbf vector tileset functionality but am unable to get working tilesets out of QGIS.

@Florian_May or others, have you created any vector tilesets yet? QGIS lets me select the shapefile(s), set the extents and zoom levels and generate the mbtiles file via "Write Vector Tiles" in the processing toolbox, but when I open the tileset it only shows a single vertical line, and also crashes out of the geowidget back to the form if I zoom in close. The file covers ~70x100m and the output is 1.3mb..

From what I can see one approach is to convert the shapefile to GeoJSON, set the CRS to 4326, then use tippecanoe to convert json to mbtiles.

Edit: mapbox doesn't like the file as an upload either, firstly it wouldn't accept zoom level >16, then "Vector tile layer has no features" (attribute table in QGIS shows 14520 features). The Z0-16 file also rendered as a single line in ODK but doesn't display the zoom/crash behaviour.

On the run just now, in brief, I believe there's no way to style a vector tile yet. There's a thread here about this issue. Would be great to have but it's a tough issue to solve.

I think my problem is simpler than that - I don't currently have a working process to create a vector tileset that will load in ODK.

I have seen the styling thread previously and currently it doesn't matter to me what colour the lines render as, as long as they're clearly visible.