Generate offline background imagery for ODK
Following on form the ODK Collect docs on offline maps, this write-up is a brief worked example of step 1 of the quickstart.
The input are freely available* raster and vector data sources plus own reference data.
(*Reminder: Adhere to the copyright and usage terms of the imagery used.)
The output is one or several MapBox Tiles (.mbtiles) files placed into a mobile device's ODK Collect layer folder at /Android/data/org.odk.collect.android/files/layers
.
Note on the ODK Collect storage location: latest ODK Collect versions are forced by a change in Android policies to store app data in the internal storage. The SD card is no longer an option. Some ODK Collect installations leave some files behind on the SD card.
TL;DR: Upgrade to latest ODK Collect, let it migrate its storage, and use the file paths from this tutorial.
Software
Quantum GIS v3.14 or higher (download from here)
Load reference imagery and data
In this step, we will prepare the imagery in QGIS exactly like we'd like to use if offline in ODK Collect. We will:
- Load some vector imagery as fallback, e.g. OpenStreetMap.
- Load some raster imagery of sufficient resolution.
Lowermost layer: vector tiles
- Layer > Add Layer > Add XYZ Layer
- If OpenStreetMap is not available: New
- URL https://tile.openstreetmap.org/{z}/{x}/{y}.png
- Name e.g. OpenStreetMap
Middle layer: raster imagery
- Layer > Add Layer > Add ArcGIS Map Service Layer
- New
- URL https://server.arcgisonline.com/arcgis/rest/services/World_Imagery/MapServer
- Name as you like, e.g. ESRI World Imagery
- Order layer above OpenStreetMap
- Lowest visible scale of ESRI World Imagery is ca 1:1050 in populated areas and around 1:1150 in less populated areas.
- Layer properties > Rendering > Maximum (inclusive) > 1:??? (lowest visible scale in your area of interest)
- Setting the maximum rendering scale ensures that this layer will be hidden when zooming in further, revealing the OpenStreetMap layer below, instead of covering it with grey "No imagery available at this zoom level" tiles.
Topmost layers: reference vectors
- Add any vector layers of your own with appropriate styling.
- Consider layer transparency, labels with drop shadow, buffer, and scale dependent visibility.
- The attached example project contains a GeoJSON layer containing one example location and styling.
Generate MBTiles
In this step, we'll generate the MBTiles file from the raster and vector layers shown in QGIS for individual locations we want to take offline.
QGIS can generate MBTiles natively, see the QGIS docs on Generate XYZ Tiles.
- Open Processing toolbox > Raster tools > Generate XYZ tiles (MBTiles)
- Zoom QGIS to each locality of interest and adjust the the raster layer's rendering maximum scale to the available scale. This step is necessary to avoid grey placeholder tiles overlaying the OpenStreetMap imagery.
- Extent: draw on canvas
- Zoom min 5, max 20 or to taste. Every additional "high" zoom level increases the resulting .mbtiles file size exponentially.
- Tile format: PNG is better quality, but ca 20 times the size of JPG at 75% quality
- Save to an output file (.mbtiles)
- Review the resulting .mbtiles by opening in QGIS (drag and drop onto layers pane). Do all zoom levels work? Is the resolution sufficient? Iterate to taste. Save "good" settings as a batch job as shown in the attached example.
The MBTiles show the imagery and vector overlays at any zoom level just as they appear in QGIS.
Including reference layers, such as "my study sites" or "my known locations" is useful as context for data capture using the manual map widgets, and to review saved submissions in situ.
Transfer and test
Transfer the .mbtiles to your devices as per docs - copy into /Android/data/org.odk.collect.android/files/layers
. Test and review raster zoom level, imagery resolution and extent. Especially getting the cut-over between raster and vector basemaps, set by the raster layer's scale-dependent rendering threshold requires some trial and error.
In forms, the maps work nicely with "manual (No GPS)" maps (appearance="placement-map"
) where users can capture a location other than their current one.
The other useful application is a background layer in the "map view" of saved forms to review already captured forms over the offline map.
Further considerations
Optimise imagery extent
There are tools available to further reduce .mbtiles size. An interesting approach is to exclude unnecessary tiles; e.g. if the area of interest is long, narrow, and diagonal (turtle nesting beaches come to mind), the square .mbtiles will contain two triangles full of images you'll never need.
Storage limits
Some OS limit the size of files you can easily transfer. Mobile data capture devices have limited storage. Our oldest devices (Lenovo Tab3) have 16GB all up, with ca 10GB available.
Running start
The attached example project contains:
- A QGIS project file (.qgz) containing:
- ESRI World Imagery as raster layer.
- OpenStreetmap as fallback vector layer.
- An example vector layer (areas.geojson) with one polygon.
- A QGIS style file (.qml) for the example vector layer.
- A saved batch job (.json) of the "Generate XYZ tiles" tool.
- The resulting .mbtiles file from the batch job.
Note: The selected area for the MBTiles export is tiny to keep the file size portable.
ODK tutorial.zip (1.8 MB)