ODK Collect v2022.2 Beta: select from map, geojson datasets

[LN]

ODK Collect betas are an opportunity to get community feedback on upcoming releases. If you have an ongoing data collection campaign, we recommend quickly verifying your form on a test device. The release will be delayed until all reported issues are fixed.

Joining the beta program
To join the beta program, find ODK Collect in the Play Store on your device (not in the web browser) and scroll all the way down. Please don't join the beta with a device or account actively used for data collection! In particular, note that joining the beta is account-based. If you use the same Google account across multiple devices, do not join the beta with that account.

Leaving the beta program
You can leave the beta program from the bottom of the Play Store at any time. Once you leave, you will get the next production update when it is released. If you need to go back to the previous production release, uninstall and reinstall the app. Your settings will be reset but your forms will remain (though backups are always recommended).

What to check in this release

• Creating projects from QR codes and reimporting from QR codes. We have implemented more validation around those QR codes.
• Downloading and updating forms with media files. It should now be fast to get a form update with media files that haven't changed. @ahblake
• Automatic form updates. Collect now backs off and retries a failed download attempt.
• Improvements to notifications triggered by auto-send.
• Pen color remembered when using image question types.
• Select one map appearance to map choice selections (aka select from map).

Additional testing
You can find a full list of changes in the release notes:

• https://github.com/getodk/collect/releases/tag/v2022.2-beta.3

Thanks to all testers for your help!

[mathieubossaert]

@LN

Downloading and updating forms with media files. It should now be fast to get a form update with media files that haven't changed

Thanks to the team !

Maybe I misunderstood the functionality, but, in "form testing / draft mode", Collect still preload every media file to run the form at first launch.
I haven't already try it on a "non draft" form.
Does the feature only impact download medias from Central to Collect ?
Which is already a good point !

[LN]

@mathieubossaert

Does the feature only impact download medias from Central to Collect ?

That's right, this is only focused on downloads.

Beta 2 is now rolling out and includes the select from map feature . It also includes improvements to notifications triggered by auto-send.

This beta is our release candidate and we are currently planning to release during the week of April 11th.

[mathieubossaert]

Thanks to the team for that work! Example with internal geometries is OK and very promising. I will try it on a real dataset this week
Example with external geojson file work and shows a classical select.

[mathieubossaert]

@mathieubossaert

Example with external geojson file work and shows a classical select.

I meant don't work, sorry for the mistake. just get a classical select appearance.

[LN]

@mathieubossaert

don't work

Whoops, it would certainly help if I put the map appearance! Let me try again.

Here are some samples for verifying the beta:

• map-internal-select-localized - defines a map with choices from within the form
• map-internal-select-advanced - defines a map with choices from within the form and shows filtering, using additional properties, etc. All of this can be done with a geojson or csv attachment as well.
• geojson-1000.zip (36.7 KB) - defines a map with choices from a geojson file. I also included a little Python script that generates a random geojson file in case you want to build a file similar to one you might use for your purposes.
• Select from osm data + sacre-coeur.geojson - defines a map with choices exported from OpenStreetMap and converted to geojson. Demonstrates using a property as a default to be updated.

This post has the most recent specification and usage. We'll be writing this up as formal documentation shortly.

For the last example, I pulled data from OpenStreetMap by opening this quadrant and clicking export in the upper left corner. I then filtered out all of the non-point features in a text editor and used https://tyrasd.github.io/osmtogeojson/ to get a geojson file. Because the OSM property used to name a feature is name, I added label=name to the parameters column of the XLSForm. This tells Collect to use the name property as the label. I also added value=id even though that's the default to demonstrate that the underlying value can also be configured. The result is that I can select an OSM feature, view all its properties, and provide an updated amenity type and name:

@Ivangayton, @danbjoseph, @Matt_Berg I want to make sure you see this as it's pretty close to supporting workflows OpenMapKit did. If the goal were to update the OSM record, a script would have to take form submissions and convert them to the OSM format. One of the challenges would be dealing with updates that might have happened while a data collector was offline (e.g. "Restaurant Délicia" gets an online update bumping it to version 3 while I'm using this form to also update it).

As I mentioned in the feature thread, we're interested in feedback on performance. We also want to hear from you how many additional properties you think you might want to include. Currently we show them all in an unstyled bottom sheet. This is likely what we'll release for v2022.2 and based on feedback we get around how people want to use this feature we'll work on how the bottom sheet looks.

[mathieubossaert]

@LN

Whoops, it would certainly help if I put the map appearance!

I should have saw it...

[LN]

Thanks to those trying the select one from map functionality! I've received some private feedback that I want to share.

First, there are some performance issues when using many points with Mapbox and Google Maps. We hope to have those fixed for this release. If you have plans to use this functionality, it would be helpful to know the number of points you expect to map at once (excluding points in the dataset that are filtered out by a choice filter).

Second, there's currently an inconsistency between how the map launches with openstreetmap and Mapbox/GMaps basemaps.

Openstreetmap launches centered on the device's location (left) whereas the other two fit all points in (right):

It's easy to toggle between the two using the buttons on the right but it would be helpful to hear whether anyone has strong opinions on what the default should be. I'm currently leaning towards current device location. Clearly both are useful for different workflows. But I think current device location will be more commonly desired.

[mathieubossaert]

Really great to be able to click on the map to select an object !!!

@LN

We also want to hear from you how many additional properties you think you might want to include

Maybe just the title and a "description" property would be sufficient on the map. I we want to show more properties, we can do it in a calculated note just after the selection ???

About performance, yes, it is quite slow to show 1000 markers on the map but does it have sense ? Maybe we could, as you discussed on Github, zoom in, close to GPS location instead of showing the whole geojson layer.
But I do not know how it works and I do not know if the problem is to printing 1000 plots on the screen or finding in the in the screen extend.

[LN]

@mathieubossaert

if the problem is to printing 1000 plots on the screen or finding in the in the screen extend.

I think we'll be able to speed both up so it's more about what's most useful.

Here are some types of usages I'm thinking about:

• the dataset clusters points in some way. It could be by something like neighborhood or even day of an exercise. The form asks some questions to filter the dataset to related points only. If I'm in the field, there might be value in me seeing all the points on that filtered map so I can mentally plan a route (keep in mind that completion won't be shown on the map). Zooming in on the current location is still valuable to show me my next move.
• the dataset has many points like the one in my screenshot above. In that case it seems there's no value to showing all points at once and zooming into the current location is much more helpful.

[Matt_Berg]

Thanks again to the team for the great work on this!

I haven't tested yet with a large number of submissions but I will and provide feedback.

For the location part - I think you want that to zoom in when you make that active.

When we implemented this in mapbox we went with the standard android location dot. Is there a reason for the crosshair? Does that make it easier to make this consistent between mapping libraries?

[LN]

@Matt_Berg

For the location part - I think you want that to zoom in when you make that active.

Thanks, will do!

@Matt_Berg

Is there a reason for the crosshair?

We made this change a couple of years ago as part of a broader refresh of the geo widgets. The reasoning is that it lets us both draw user attention to the device position with a fixed visual (crosshairs) and show the accuracy radius as a semitransparent circle. This was arrived at following some field studies by @Ivangayton and @zestyping. More in the issue. My screenshot above doesn't have the inner circle because I was spoofing my location with an accuracy of 0. It certainly does look better with the inner circle.

Does the dot visualization you reference provide different information? Does it indicate accuracy or is it of a fixed size?

[Ivangayton]

Well, this is fantastic!! I spent most of today testing.

First reflections:

• When you first open the widget, the zoom level spans the entire data set. My intuitive expectation is to start from a fairly high zoom level centered on my own location, but I guess the current behaviour is reasonable since it's fairly obvious that you just need to hit the my location icon to zoom into where you are.

  • However, when I hit the target icon to go to my own location, the zoom level is fairly low, and my target location is not centered. I invariably find myself zooming in more.
  • Update: this is actually upsetting me more as I proceed with testing! Once I get into a flow, I'm visiting lots of features in the same area. Every time I launch a new form and come to the select from map question, I have to hit the icon to pan&zoom to my location, and then zoom in some more before I can effectively select a point.
    • I think this behaviour makes perfect sense when you're just testing the functionality on a single point (or even when you visit your first point, at which moment it's probably nice to see where you are with respect to the whole dataset), but once you're spending a significant amount of time mapping multiple points, those two extra UI interactions per feature are aggravating.

• The GeoJSON is pretty unforgiving! I've created samples a few different ways:

  • Download OSM data from Overpass, import it into QGIS and convert polygons to centroids, export as geojson from QGIS
  • Download OSM point-only data from Overpass directly as geojson
  • Manually create QGIS point layers and export them as geojson
  • Handwrite geojson

• No matter how I do it, it usually takes me a few tries to get the geojson working. I'm not even sure what the problems are: right now I'm mostly making them work by deleting almost everything except the exact points and attribute fields I want. Sometimes the error messages are helpful (it seems that if I make a simple syntax error in the geojson it gives me a line number and descriptive error) but other times they are terribly opaque, for example An unknown error has occurred... Attempt to set Uncast Data value to null1|AnswerData objects should never have null values. (maybe this is when fields that the form attempts to perform calculations on are empty or null?)

• I think it would be fairly common to wish to add new points, in the same way that OpenMapKit did. In other words, if the feature exists in the real world, but not in the GeoJSON, you might wish to create a point and then fill out the same form, albeit without the pre-populated fields.

  • I suppose a workaround could be using an XLSform that includes a Geopoint question that's relevant if there's nothing selected from the map.
  • Yet another workaround would be to have a separate form that asks all the same questions but simply replaces the select from map with a geopoint. But you won't know that you need that until you've already tried to select the relevant point from the file.

• Side reflection: is there anything preventing us from using the same media dialogue that we use for the GeoJSON upload to also optionally provide a raster base map? It seems like a matched set of a GeoJSON and a raster base map (and eventually a vector tile overlay) would go together nicely. Something particularly cool about this functionality that I hadn't really thought about before is that you're pretty explicitly defining an area of interest when you create the geojson file. Therefore it seems like a great starting point to add base layers from the Central server.

[LN]

@Ivangayton

I spent most of today testing.

We released v2022.2.0 on Apr 13. Is that the version you were trying out or was it still the beta? I realize now I should have made sure to announce the release in this thread. Next time!

@Ivangayton

When you first open the widget, the zoom level spans the entire data set.

One change we made from the last beta to the release is to zoom to current location on load. Please give that a try if you haven't already. One thing I am not entirely satisfied by is that acquiring location can take a moment and so the view can change once I've already started interacting with it.

@Ivangayton

my target location is not centered

I can't reproduce this. Is it maybe that it first centers on an inexact location and then gets a new location? A screenshot here might help us figure out what's going on.

@Ivangayton

I have to hit the icon to pan&zoom to my location, and then zoom in some more before I can effectively select a point.

Could you please share a screenshot? I'm interested in seeing the basemap that you are using, the size/resolution of your device, and the density of features you have to select from. If you haven't already, can you try this with an OSM basemap (https://docs.getodk.org/collect-settings/#maps-settings)? There's still some variability between the three engines we use and I'd say OSM is the best-behaved at the moment.

I think this intersects with Geo: Optimizing zoom levels for walking surveyors as well. Some of these zoom level considerations are going to be pretty domain-specific. It sounds like you're in a dense urban area. If the goal is to navigate to and select a field or a campground, the appropriate zoom level will be different. Can you share some example of datasets you want to use?

@Ivangayton

The GeoJSON is pretty unforgiving

Can you share forms and GeoJSON files you've tried to use? I have a guess as to what that Uncast Data error is about but I'm not totally sure.

@Ivangayton

using an XLSform that includes a Geopoint question that's relevant if there's nothing selected from the map.

That's how I would do it currently. I would also write instructions to that effect in the question label so that they show up at the top of the map screen.

@Ivangayton

Therefore it seems like a great starting point to add base layers from the Central server.

Currently custom offline layers exist and are configured at the project level, not at the form level. So there would have to be some thinking around how that's harmonized. Relatedly, Central only requests/accepts attachments that are explicitly referenced in the form and there's no form spec for declaring anything about custom map layers. None of this is insurmountable but it requires spec design work.

[chun_hing_yap]

I think 'the select from map' using geojson and 'viewing filled forms on a map' is actually unrelated and independent feature. The former is a choice selection via map approach and the latter is map visual for individual form submission.

I am using odk repeat quite often with repeat group feature, eg. recording repeating tree many times in a single form. I think the 'select from map' should work well for repeat group though I have not evaluate it yet.

And, I am thinking whether there is a way to display the map in 'select from map' where the the map can distinguish the marker with records and marker without records. This will be a great help for the data collector to identify the marker which have yet to complete for the data entry.

If this can be accomplished. It will complement the inability of 'viewing filled forms on a map' feature for geopoint in repeat group.

[mathieubossaert]

Hi @chun_hing_yap

Here is a example for select_one with map appearance (also relevant for the question you asked here)

You can achieve it with a form "session" using the choice filter described in this post by @LN

I needed the same for an exhaustive inventory of 100m squares covering a coastal lake.

As @Ivangayton and @LN suggested here, we use two select_one with map appearance :

• the first one hides already visited squares (centroids)
• the second one all the grid.

Prefer OSM map renderer in your project preference to test.
Squares are located in South of France.
select_one_map_hide_already_selected.xlsx (12,2 Ko)
mailles_100m_etang.geojson.zip (22,1 Ko)

[Ivangayton]

Minor issue with geojson created by QGIS; my default process ofr creating a geojson file of points from scratch in QGIS creates a layer with some attributes, like so:

{
"type": "FeatureCollection",
"name": "things_with_layer_attributes",
"crs": { "type": "name", "properties": { "name": "urn:ogc:def:crs:OGC:1.3:CRS84" } },
"features": [
{ "type": "Feature", "properties": { "gid": 1, "visited": "yes" }, "geometry": { "type": "Point", "coordinates": [ 130.559803472160752, 31.580360584438704 ] } },
{ "type": "Feature", "properties": { "gid": 12, "visited": "no" }, "geometry": { "type": "Point", "coordinates": [ 130.560375586896328, 31.579278707364338 ] } }
]
}

This fails to load, with message

problem found at nodeset: /html/head/model[@xforms-version=1.0.0]/instance
with element <instance id="things" scr="jr://file/things.geojson"/>

Manually deleting the layer attributes and proceeding straight to the feature array (as below) avoids the problem; this GeoJSON works:

{
"type": "FeatureCollection",
"features": [
{ "type": "Feature", "properties": { "gid": 1, "visited": "yes" }, "geometry": { "type": "Point", "coordinates": [ 130.559803472160752, 31.580360584438704 ] } },
{ "type": "Feature", "properties": { "gid": 12, "visited": "no" }, "geometry": { "type": "Point", "coordinates": [ 130.560375586896328, 31.579278707364338 ] } }
]
}

[Ivangayton]

The GeoJSON parser seems to freak out if an attribute in a geojson is null, like

"name:ja": null,

Replacing all unquoted nulls with a pair of double-quotes avoids the problem, this works:

"name:ja": "",

So for now if you have attributes in the GeoJSON that are not filled out for all features, you might need to replace the nulls with empty strings.

Additionally (I think this one is already known; pardon me if I'm repeating already-reported issues): if the list of attributes is too long, the Select button at the bottom of the list goes off the bottom of the screen and can't be reached. So I trim the number of attribute fields.

Here's a working GeoJSON that I created by:

• Using Overpass Turbo to download buildings with query:

[out:json][timeout:200];
(
  nwr["building"]({{bbox}});
);
out body;
>;
out body;

as GeoJSON.

• Then pulling the polygons from the JSON into QGIS, creating centroids, and saving as GeoJSON with only a few fields (id, name, amenity, building, etc).

• Then manually editing the GeoJSON to remove the layer attributes (as per previous post; removing everything between "type": FeatureCollection", and "features": [.

• Then global replace-string of null with "".

And voila, it works with the (extremely basic) xlsform attached.

map_osm_buildings_in_odk.xlsx (9.3 KB)

buildings.zip (6.8 KB)

[LN]

@chun_hing_yap

I think 'the select from map' using geojson and 'viewing filled forms on a map' is actually unrelated

They're related in a context where a form submission represents a single encounter with an entity (person, place, thing). The filled forms map then represents completed work. Because you use repeats to represent your entity encounters, the two indeed don't feel as related. In general, there are tradeoffs between using one submission per entity vs one repeat instance per entity but most scenarios can use either. The one that makes the most sense will depend on data collection and analysis workflows.

@chun_hing_yap

And, I am thinking whether there is a way to display the map in 'select from map' where the the map can distinguish the marker with records and marker without records. This will be a great help for the data collector to identify the marker which have yet to complete for the data entry.

I talked a bit about this in the feature thread and described strategies for cases where one submission represents one entity encounter. @mathieubossaert's example above uses a similar strategy but with repeats. It's a great example!

@Ivangayton

creates a layer with some attributes

@Ivangayton

freak out if an attribute in a geojson is null

Thanks for both of those! We put in a bunch of flexibility and tests at the Feature level but were indeed having unrealistically specific expectations at the top level. I have https://github.com/getodk/javarosa/pull/681 to fix both and I think we'll do a point release for it. If anyone else has geoJSON files that are getting rejected, please share or describe them.

If you have thoughts on any of my other follow-up questions above, we might be able to get some of your other issues resolved in that point release.

@Ivangayton

name:ja

Is this a localization convention that is documented somewhere? We have been thinking about localized labels.

@Ivangayton

if the list of attributes is too long, the Select button at the bottom of the list goes off the bottom of the screen and can't be reached.

Indeed! We've left the bottom sheet unstyled pending feedback on how users want to use it. How many attributes? What screen sizes? Any information on how it would be used in an ideal world would help us make decisions there.

[mathieubossaert]

I faced the same troubles than @Ivangayton with QGIS ("header" and null values) and he was faster to post about. I did not yet investigate export options to see if it could solves the problem on QGIS side.