Provide a way to get mbtiles to Collect without having to manually place files in the layers directory

[mathieubossaert]

@Ivangayton

If this sounds useful for your farm mapping, let me know, I'd be delighted to set up an FMTM task with you to see if it is useful for your team.

Thanks Ivan for your offer ! I really appreciate
I know the impressive work on FMTM, but I think it can't help us, in our context.
And for the moment we are not stuck. I am just thinking about how this new feature we are discussing may help us in the future.
Farms mapping is an example, but the reality is more complex.
I do have 50 colleagues using ODK on the field.
They each use 5 to 10 of the 30 forms we host (depending on their skills and missions), over 1 one to 50 of the 300 sites we study

Our dataflow relies on cron calls of Postgresql fonctions on each form. And I don't want to make it more complex with too many projects to pull data from.

[seewhy]

This is a really helpful discussion from my perspective.

I think as 'proof of concept' the use of a URL to 'grab' a basemap is a great step forward - although it is not entirely intuitive, one of its strengths is not disrupting the existing workflow... In my opinion. And if it is possible to grab a series of MBTiles (for example) via URL that could be useful as part of pre-data collection preparation, then just select the appropriate basemap in the form.

I confess to being nervous of the external TMS approach - that's another learning curve, potentially an account or server set up and maintenance / cost, and I'm not sure yet if it would give me the required basemaps in all situations.

Two use cases to help me think it through:

Field validation of work completed
We use a custom basemap with (at least) 3 vector layers overlain on a topographic backdrop (or sometimes drone derived orthomosaic). The data in the vector layers may be confidential. I have no need (or desire) to have any other those vector layers selectable and to date I have produced raster MBTile layers using QGIS. This results in some fairly large files if we're working on an extensive site, but as I plug the device into the computer this is not a problem. We are a small team with only 1 or 2 people remote from this computer - generally we share the file(s) via our webserver or any of the (free) filesharing services and provide instructions (devices are not locked). This fieldwork (re)uses a single form at multiple locations and is ongoing - this means that defining / linking a specific basemap to the form would require repeated intervention via Central and therefore more coordination - we sometimes work on 3 or 4 different sites in a single trip away from the office, and may not have reliable data connection whilst away. Each member of the team is able to create and load their own MBTiles basemaps (but don't always)

The file picker would work well providing we can get the files on to each device easily (we usually create a sub-folder within Downloads to store GPX tracks, so would probably implement something similar via a training session). Presumably feature this would be accessible from the form geo-widget as an extension of / alternative to select basemap? Which is more intuitive than needing to access the project settings...

External data collectors
Using locked devices, up to 40 remote staff, not under my management: I've explained this elsewhere I'm sure, but basically we have custom basemaps (probably more akin to the Area of Interest model) for defined sites and need to share these without being able to plug in the device or access the Android filesystem. With our own devices lent to the client (and mailed between sites!), we store all the basemaps on the device and select the appropriate one for each location within the geo-widgets. We also have point / line data for each of these locations which we now use select-one-from-map, with a filter to load a subset, based on the location. In a previous iteration of the form, we included these features on the basemap, but that is probably not essential.

Again the file picker would work (we already use this to import GPX tracks to the form prior to sending, so the concept would be familiar). Although some sites have data connection, most would need to prepare ahead of doing fieldwork by downloading the basemap.

I've just looked at my 'master' folder on the desktop and have around 400 basemaps from the last 3 years, taking up just under 3GB. Two projects with 50+ basemaps that were generated in bulk, otherwise they were created singularly for specific locations / projects, using any one of 5 forms...

I feel like I'm oversharing here! But hopefully it gives an idea of what functionality might be required - something relatively simple with a short workflow would be good

[Nishon]

Hey! all, thanks for the vibrant discussion.
It made me realize there is a need for this.

So, I have made another version that adds a file picker option in the layer selector, and you can use it to load any MBtile stored on your device.

Here's a screenshot of the workflow.

Here's the link to the downloadable app and the source code

Release Enhanced .mbtiles File Import Functionality in ODK Collect ·...

[LN]

Thanks so much for trying this alternative, @Nishon! What do you think, would it meet your needs? Is there still one you clearly prefer over the other?

Our current plan is to add one way to get mbtiles to Collect in the following release of Collect which will come out in December or January. We want to do just a bit more user research and design work before we finalize which approach to take and then should be able to use one of your implementations as a starting point. How does that sound?

[Nishon]

@LN

We want to do just a bit more user research and design work before we finalize which approach to take and then should be able to use one of your implementations as a starting point. How does that sound?

I request access to user research findings.
How can I contribute to this being a PR, ensuring the solution is not narrow and considers edge cases (especially around the Android life cycle), tests, and documentation?
Additionally, I'm open to a call with a core dev to ensure code alignment with Collect's standards.

[LN]

@Nishon

I request access to user research findings.

Yes, we can synthesize this thread and any other conversations we have soon.

Most importantly, now that you've built this latest demo, does it feel like the file picker approach would work for your needs? I realize it's not quite as convenient for those needs as what @Ivangayton described in an earlier comment. I do think it has a lot of nice properties that have been described in this thread.

I also see

@Ivangayton

I'd be happy with either for now; both would immediately allow most of the people I know and work with to use MBTiles basemaps relatively easily.

[Ivangayton]

Certainly works for me.

[Aly_Blenkin]

Hi everyone - we have some updates to share!

We had a call with the HOT team this week, and everyone is aligned on the file picker and device storage. We've also iterated and refined the user experience to hopefully meet everyone's needs. Here is a mockup to give you an idea of the direction and what we are going to start building

Key design considerations

• Information hierarchy: clear titles, descriptions, and CTAs so users understand what they are looking at and where to go if they need more information.

• Add/delete MBtiles layers: We included the ability to delete because the layers could be large. We are still unsure if delete is critical for the first iteration.

• Keeping the path: Without an option to delete there would be more manual management. We decided to keep the path in the UI because titles could be the same or similar, and the path could have helpful information (e.g. determining if the layer is shared between projects or only at the project level). It also means users could use adb to connect from a computer if they want. By hiding the paths, it makes it easier to scan if you have multiple layers to choose from.

• Navigate via map or settings: the same functionality would be in both spots, although we assume discoverability will be higher from the map question.

Feedback

• Terminology: in the mockup we use the term "reference layer", but we've also been considering "MBtile layer" or "offline layer" as alternative options. What do you think would be the most clear for data collectors?

• We are curious to know what you think. Let us know in the thread, or give us a thumbs up if you like the direction.

[spwoodcock]

This looks fantastic & covers most cases well!

I love the confirm deletion dialog.
On the implementation side:

• If the file is stored in a project specific directory, then it's easy to determine the mbtiles are associated with a project.
• But if an mbtiles file is associated with many projects, I wonder what the mechanism is to determine this.

As for the layer terminology: it seems to me that the mbtiles are exclusively used for basemaps, so something like 'Offline Basemaps' could work.
Otherwise, I like 'MBtile layer' best from the options, as it's the most descriptive.

[LN]

@spwoodcock

if an mbtiles file is associated with many projects, I wonder what the mechanism is to determine this

I think the easiest way will be to iterate through shared prefs for each project. Currently there is a single setting for reference layer per project.

@spwoodcock

it seems to me that the mbtiles are exclusively used for basemaps

PNG tiles can have transparency so it is possible to use an mbtiles file as a layer on top of an online basemap. It’s also possible to specify vector (pbf) tiles which would typically be more sparse, especially because currently there’s no way to style vector mbtiles. Because of the style limitation I doubt it’s used much.

My recollection is that the term « reference layer » came about when we thought vector mbtiles were viable with styling. We also had the idea that we might allow geojson reference layers. If any of these still seem compelling we may want to keep the name somewhat generic so we don’t have to change it again.

[spwoodcock]

That's a very good point.

'Reference layer' does encompass more & is probably the most versatile term. But I think it is a little ambiguous for users.

'MBtiles layer' is good for developers, but I should keep in mind that is not the main audience.

Personally I think 'Offline layer' is the best tradeoff between accessible language and scope of usage.

[LN]

@spwoodcock

But if an mbtiles file is associated with many projects, I wonder what the mechanism is to determine this.

@seadowg pointed out that in a workflow where people are switching between different offline layers the information about which projects a layer is actively used with might not be useful or could even be misleading. Maybe we remove that extra complexity for now?

@yanokwa also mentioned to me that the file size might be something helpful to include in the confirmation dialog. For example, you might really want to make sure you're done with a very large file before you delete it because it could take a long time to download again. I don't think this needs to be in the first version but I do think it's an interesting idea.

@spwoodcock

Personally I think 'Offline layer' is the best tradeoff between accessible language and scope of usage.

That's sounding reasonable to me! If/when we do allow for layering on custom online map data for reference, we can decide whether it combines with this existing functionality or is introduced as a separate concept.

@rsavoye any thoughts on your end? @mathieubossaert @rudo and @seewhy I think you likely also have interesting perspectives on both the latest design and the language.

[rsavoye]

I've not seen a imagery basemap used in multiple projects, although it's entirely possible that could happen. I agree I don't think you need to track which projects a layer is active in. For me what's more common is multiple basemaps for each project. Often I'll toggle between USGS topo maps and different imagery.

Offline layer works for me too, since basemap is used elsewhere in Collect.

[seewhy]

This is looking really good - I am happy with the file picker approach, great work adding that functionality @Nishon and the mock up is excellent @Aly_Blenkin .

I like the ability to see the file path if required, but it being collapsed to help navigate with multiple files.

Ability to delete is important too, so I like that feature. I concur with @yanokwa about issues of file size - also useful if you have old ones cluttering up a device...

Generally my projects dump layers in the shared folder, but that was mainly to avoid even more complexity (I have had the phone call from the enumerator in the field who can't find the reference layer they swore they had uploaded to the 'layers' folder...).

Personally I'm happy with the term Offline Layer as it is generic enough to cope with future opportunities (I can hear the Core Team groaning already - "we haven't even implemented this one yet!). It also 'contrasts' well with Online layer so helps to avoid confusion!

It's useful to be able to switch layers in the form, via the map, so being able to add additional layers from storage at the same point makes a lot of sense - not every enumerator prepares before starting the form! Ahem, speaking from experience...

I'd put 3 thumbs up if I had that many! Thanks for your work on this.

[mathieubossaert]

@LN

@mathieubossaert @rudo and @seewhy I think you likely also have interesting perspectives on both the latest design and the language.

Hi @LN, as you can imagine, my silence does not reflect our interest for this great promising feature .
Thanks to both HOT and ODK teams for the work and thanks @Aly_Blenkin to make it visible before it existed

I could copy/paste @seewhy 's response as it suits me fine !

About the vocabulary, "offline layer" is generic enough to be reusable in future features, with raster or vector layers (tiles or not).

I imagine to use it in combination with our nextcloud, to assist the user to transfer the files from the computer to the phone.

[rudo]

Hello! Apologies for a delay as I was offline for a month with family. But excited to see these mockups and to see the offline map work progressing

I do agree that “offline layer” is an improvement on both “reference layer” and “MBTile layer.” Reference layer is a bit too specific, in my opinion, whereas MBTile is not only an overly technical term but risks becoming outdated as new map tile formats (such as pmtiles) become more widespread and we may want ODK Collect to support their usage.

I was wondering why not use “offline background map” or “offline basemap”, if it’s only ever possible to use one single map tileset (mbtiles file)? “Layer” gives me the impression that I could select multiple tilesets for my map view. Also, map tilesets frequently combine a multiplicity of different layers (e.g. land use, roads, points of of interest), so that could be a bit confusing. That is certainly true of vector, but a raster mbtiles can also have multiple layers baked into one rasterized image (for example, any of the above overlaid on top of satellite imagery scenes).

That said, I do think most users will understand what is meant by "offline layer".

[LN]

Thanks for the feedback, everyone!

@rudo

why not use “offline background map” or “offline basemap”

Some people use this functionality to overlay some custom data on top of the online basemap. Collect always tries to render the online map and then stacks whatever the mbtiles files defines on top. If the tiles in the mbtiles have transparency (raster) or are sparse (vector), the online basemap will show through (or gray if offline).

That's how we ended up with the term "reference layer" because it's a layer (possibly composed of multiple layers as you say) that has custom reference data. But of course the probably more common usecase is to specify a tileset that is opaque in which case it does act like a basemap. We wanted to be precise with the naming but I think it's just pedantic so I'm glad we're coming up with alternatives.

@rudo

map tilesets frequently combine a multiplicity of different layers (e.g. land use, roads, points of of interest)

I feel like I've seen multiple flattened/combined layers referred to as "a layer" as well. But maybe not? We could possibly consider "offline tileset"? "Offline tiles"? Or it sounds like "offline layer" might be clear enough for now and then we could revise it as we expand what custom map data can be specified.

[rudo]

Collect always tries to render the online map and then stacks whatever the mbtiles files defines on top. If the tiles in the mbtiles have transparency (raster) or are sparse (vector), the online basemap will show through (or gray if offline).

Ah, I didn't know that! That is neat. Now I'm fully convinced that "offline layer" is a great name for this

[ahblake]

This mockup looks good to me and should be easy to train enumerators to use. (Currently I have them copy-pasting with the device connected to a windows machine over USB after navigating to the correct folder)

It looks like you can nav to internal/external storage, so if I had a bulk set of tiles on an SD card, that could be connected with a USB dock and they could select/copy from there. (But they could also have downloaded them / received and saved an email attachment)

The mockup shows selecting one to many - would there be a select all option for a folder content? eg there's a '2023 maps' folder, with folders for 'location a', 'location b' etc, and in each folder there are multiple tilesets. Rather than having to check each, could either the folder be selected, or a top level 'select all' when opening the folder. (In my particular use case, for a project, there are tens of locations, each with tens of map tilesets, so to complete work at one location you need to load all of the tilesets)

I like the option of having tiles limited to a particular project, on my test device it's a mess with many tiles for many different projects all visible when selecting the desired tileset.

Being able to add tiles from the map widget as well as through settings is also a nice UI help for the user.

Offline layer as the name works for me too.

I'm dreaming a little here but is there also consideration for the form being able to dynamically set the offline layer?

[Aly_Blenkin]

@ahblake

This mockup looks good to me and should be easy to train enumerators to use.

Hey Andrew, thanks so much for your feedback!

@ahblake

The mockup shows selecting one to many - would there be a select all option for a folder content? eg there's a '2023 maps' folder, with folders for 'location a', 'location b' etc, and in each folder there are multiple tilesets. Rather than having to check each, could either the folder be selected, or a top level 'select all' when opening the folder. (In my particular use case, for a project, there are tens of locations, each with tens of map tilesets, so to complete work at one location you need to load all of the tilesets)

I can certainly see how a select all option would be very helpful, but I'm not sure how difficult it is to implement - what do you think @seadowg @Nishon? We can certainly look into it and see where it fits in our roadmap.

@ahblake

Offline layer as the name works for me too.

Seems like 'offline layer' is resonating with people!

@ahblake

I'm dreaming a little here but is there also consideration for the form being able to dynamically set the offline layer?

Can you tell us more about your use case or how you would imagine this working?