Generate offline background imagery for ODK Collect (MBTiles)

Generate offline background imagery for ODK

Following on form the ODK Collect docs on offline maps, this write-up is a brief worked example of step 1 of the quickstart.

The input are freely available* raster and vector data sources plus own reference data.
(*Reminder: Adhere to the copyright and usage terms of the imagery used.)

The output is one or several MapBox Tiles (.mbtiles) files placed into a mobile device's ODK Collect layer folder at /Android/data/org.odk.collect.android/files/layers.
Note on the ODK Collect storage location: latest ODK Collect versions are forced by a change in Android policies to store app data in the internal storage. The SD card is no longer an option. Some ODK Collect installations leave some files behind on the SD card.
TL;DR: Upgrade to latest ODK Collect, let it migrate its storage, and use the file paths from this tutorial.

Software

Quantum GIS v3.14 or higher (download from here)

Load reference imagery and data

In this step, we will prepare the imagery in QGIS exactly like we'd like to use if offline in ODK Collect. We will:

  • Load some vector imagery as fallback, e.g. OpenStreetMap.
  • Load some raster imagery of sufficient resolution.

Lowermost layer: vector tiles

Middle layer: raster imagery

  • Layer > Add Layer > Add ArcGIS Map Service Layer
  • New
  • URL https://server.arcgisonline.com/arcgis/rest/services/World_Imagery/MapServer
  • Name as you like, e.g. ESRI World Imagery
  • Order layer above OpenStreetMap
  • Lowest visible scale of ESRI World Imagery is ca 1:1050 in populated areas and around 1:1150 in less populated areas.
  • Layer properties > Rendering > Maximum (inclusive) > 1:??? (lowest visible scale in your area of interest)
  • Setting the maximum rendering scale ensures that this layer will be hidden when zooming in further, revealing the OpenStreetMap layer below, instead of covering it with grey "No imagery available at this zoom level" tiles.

Topmost layers: reference vectors

  • Add any vector layers of your own with appropriate styling.
  • Consider layer transparency, labels with drop shadow, buffer, and scale dependent visibility.
  • The attached example project contains a GeoJSON layer containing one example location and styling.

Generate MBTiles

In this step, we'll generate the MBTiles file from the raster and vector layers shown in QGIS for individual locations we want to take offline.

QGIS can generate MBTiles natively, see the QGIS docs on Generate XYZ Tiles.

  • Open Processing toolbox > Raster tools > Generate XYZ tiles (MBTiles)
  • Zoom QGIS to each locality of interest and adjust the the raster layer's rendering maximum scale to the available scale. This step is necessary to avoid grey placeholder tiles overlaying the OpenStreetMap imagery.
  • Extent: draw on canvas
  • Zoom min 5, max 20 or to taste. Every additional "high" zoom level increases the resulting .mbtiles file size exponentially.
  • Tile format: PNG is better quality, but ca 20 times the size of JPG at 75% quality
  • Save to an output file (.mbtiles)
  • Review the resulting .mbtiles by opening in QGIS (drag and drop onto layers pane). Do all zoom levels work? Is the resolution sufficient? Iterate to taste. Save "good" settings as a batch job as shown in the attached example.

The MBTiles show the imagery and vector overlays at any zoom level just as they appear in QGIS.
Including reference layers, such as "my study sites" or "my known locations" is useful as context for data capture using the manual map widgets, and to review saved submissions in situ.

Transfer and test

Transfer the .mbtiles to your devices as per docs - copy into /Android/data/org.odk.collect.android/files/layers. Test and review raster zoom level, imagery resolution and extent. Especially getting the cut-over between raster and vector basemaps, set by the raster layer's scale-dependent rendering threshold requires some trial and error.

In forms, the maps work nicely with "manual (No GPS)" maps (appearance="placement-map") where users can capture a location other than their current one.
The other useful application is a background layer in the "map view" of saved forms to review already captured forms over the offline map.

Further considerations

Optimise imagery extent

There are tools available to further reduce .mbtiles size. An interesting approach is to exclude unnecessary tiles; e.g. if the area of interest is long, narrow, and diagonal (turtle nesting beaches come to mind), the square .mbtiles will contain two triangles full of images you'll never need.

Storage limits

Some OS limit the size of files you can easily transfer. Mobile data capture devices have limited storage. Our oldest devices (Lenovo Tab3) have 16GB all up, with ca 10GB available.

Running start

The attached example project contains:

  • A QGIS project file (.qgz) containing:
  • ESRI World Imagery as raster layer.
  • OpenStreetmap as fallback vector layer.
  • An example vector layer (areas.geojson) with one polygon.
  • A QGIS style file (.qml) for the example vector layer.
  • A saved batch job (.json) of the "Generate XYZ tiles" tool.
  • The resulting .mbtiles file from the batch job.
    Note: The selected area for the MBTiles export is tiny to keep the file size portable.
    ODK tutorial.zip (1.8 MB)
3 Likes

Hello, for some reason my layer files can't be located. I've followed your process but when I try to reference the layers it comes up with the message
There are no layer files in /...................

Can you help?

Regards

Hi @dominicgusah,
Welcome to the forum! If you have a moment, introduce yourself in the welcome thread!

What ODK Collect version are you running? The tutorial assumes the latest version, which changed the internal file path.

What file path are your .mbtiles files under on the tablet?

Can you open the .mbtiles in QGIS?

Thank you, Will do.

I'm using v1.28.4

I didn't realise there was a similar 'Layer' folder in Internal storage. I was using the one in the SDcard. When I saved the files to the internal folder it worked. This is despite the Reference setting saying "SDcard/......."
How do I now change Collect to look for the MBTiles in the SDcard and not on the internal storage? (As there is usually more space there)

Regards

Ah, understood, thanks for the clarification!
The issue you're experiencing is that the latest ODK Collect versions are forced by Android to migrate their storage to the internal storage. The SD card is no longer an option for app data unfortunately, which to my knowledge is an Android limitation.
So, the paths in my tutorial are correct for your version (1.28) and we should update the ODK docs https://docs.getodk.org/collect-adb/#collect-directory.