Ability to load locations into the geowidget to guide data collection

[LN]

The Entity-Based Data Collection Working Group was formed, met, came up with a high-level project plan, and is now seeking funding. Geo will eventually be central to what we have in mind but is currently looking like it will be layered on top of other upfront work. In the meantime, Ona has expressed renewed interest in what @Matt_Berg described above.

We propose adding a map appearance and geojson support to select_one_from_file. Initially the map appearance would only work with geojson files but we could expand it to work with all select_one variants (see extensions below). select_one_from_file with a geojson file and without the map appearance would provide a text interface identical to how select_one_from_file works today.

The user experience provided by the map appearance would be similar to a geopoint with the placement-map appearance. In the form, question text and a button with text “Select place” would be displayed. If a feature has previously been selected, its label (from the name property, see below) would be displayed.

When the “Select place” button is tapped, a map view would be launched. All features from the geojson file would be displayed on top of the basemap and any reference layer configured on the device. Instead of being able to select an arbitrary location, the user would be able to tap on any feature defined in the geojson file.

geojson files used with select_one_from_file:

• MUST have a .geojson extension (not .json)
• MUST define a single top-level FeatureCollection
• MUST define a property called id for each feature in the collection (features without this property will be silently ignored; see extensions below for more flexibility which could be later introduced)
• MAY define a property called name for some or all features in the collection which will be used as the feature’s label (Note: I propose name because I’ve seen that in several geojson file examples. An alternative would be title which is specified by simplestyle. interested in opinions here!)
• MAY define any number of additional custom properties for some or all features which will be displayed when a feature is selected (and eventually available to form logic, see extensions)

Points would be displayed as markers like the ones used for the filled form map . Additional feature types could be supported as described in the extensions below.

All of the features would be of the primary app color (blue-green) when unselected. The feature selected would be displayed in red to match the existing geo widgets. In the case of points, the marker would also increase in size (same as the filled form map).

When the quick appearance is specified in addition to the map appearance, tapping on a feature would select it immediately and the map screen would close.

When the quick appearance is not specified and a feature is tapped, a bottom sheet would be shown as on the filled form map. If the feature has a property called name, that property’s value will be shown as its label (above the horizontal rule). If there is no such property, the id/underlying value will be displayed as the label. Any properties in addition to id and name will be displayed in a table below the label.

When a feature is selected and the save icon is tapped, that feature’s id/underlying value would be saved as the question’s value, overwriting any previously-saved value (same as geotrace/shape). When the trashcan icon is tapped, the current selection is abandoned and any previously-saved value remains (same as geotrace/shape).

When a geojson file is used without the map appearance, the id property would be used as the underlying value/name and the name property would be used as the display value/label.

An additional requirement is the possibility to specify a feature not defined in the geojson file. For now, we propose pushing that to form design: the form definition could include label text that prompts data collectors to leave the question blank if the feature isn’t on the map. If the value is blank, a series of questions could be asked to define the new feature. Once we have the basic requirements in place, we’ll try this and see if it’s sufficiently clear.

Extensions:

• Add feature types
  • MultiPoint, LineString, MultiLineString as outlined circles with a white center separated by lines as for the geotrace widget
  • Polygon, MultiPolygon as outlined circles with a white center separated by lines as for the geoshape widget with the addition of a light shading of the enclosed area
• Any property defined in the geojson file could be accessed in expressions the same way that values from other datasets are. For example, if a field called the_road has type select_one_from_file roads.geojson in an XLSForm, the repair_state property of the road selected could be accessed with expression instance('roads')/root/item[id=${the_road}]/repair_state. This would make it possible to do things like constrain the features that can be selected based on the current location or filter the features shown based on some condition (e.g. only show hospitals that have some specialized team)
• Add flexibility to the geojson requirements:
  • if there is no property called id, look for an id foreign member (sibling of type: https://datatracker.ietf.org/doc/html/rfc7946#section-6.1)
  • allow specifying the property to use as the underlying value in the parameters column of the XLSForm (e.g. name=the_id)
  • allow specifying the property to use as the label in the parameters column of the XLSForm (e.g. label=the_label)
• Support map appearance on select_one. This would also require specifying either lat and lon columns or a geojson column from the parameters column of the XLSForm. (XForms spec might be weird; additional children for itemset? This is why we propose introducing the map appearance rather than only using the filename extension)
• Support map appearance on select_multiple_from_file/select_multiple
• Allow styling using the simple style spec. This would make it possible for the geojson file to specify things like color for individual features.

[danbjoseph]

I'm super excited to read this. Two initial trains of thought. I'm sure there will be more. Please excuse the rambling as I work through my thoughts.

---

[1] I'm trying to think through how I would get survey answers back into the geospatial dataset. That is, if I have a FeatureCollection of buildings. And they all have some assortment of the additional custom properties (outside of the must have id). If a feature has building=yes + shop=deli and then my survey data is that actually shop=cheese + also levels=3 then how do I get my output FeatureCollection that has that feature with all three of the right tags building=yes; shop=cheese; levels=3. This is probably a matter of an external process/script that merges the data spreadsheet download from Central with the source GeoJSON?

The scenario where we may only be adding details, like a damage assessment where we are adding a damage_level=* custom property that no feature has yet, seems pretty simple in then getting back to your desired output for analysis (basic join of a survey data table to the FeatureCollection in a GIS software on the id field).

A mapping scenario in which we might be adding/ editing/ removing custom properties, seems more complicated. If someone clicks a feature with amenity=place_of_worship then we have a questions asking them if they can add or update religion=* and denomination=*. If a feature has shop=* then we ask if they can add or update name=*. And so on... and then we want to bring together the starting FeatureCollection data with the assortment of new/ changed/ removed attributes.

---

[2] When would a user need to see the vertices of a polygon/multipolygon? The geoshape widget in ODK is for editing the shape and so you need to know where the points are. In the OSM iD editor it shows the points when a shape is selected for editing. But otherwise it just shows the border and the translucent fill.

[mathieubossaert]

Was it my birthday yesterday ?...
Yes it really was

[yanokwa]

@LN

Any property defined in the geojson file could be accessed in expressions the same way that values from other datasets are.

I believe this would enable using either a choice filter to filter out features based on a previously-captured location, right? So you could require someone capture their location, then once they do that, they only see features that are say 20 meters of their location?

Edit: Yes because I see now that you also said...

@LN

This would make it possible to do things like constrain the features that can be selected based on the current location

[LN]

@danbjoseph

I'm trying to think through how I would get survey answers back into the geospatial dataset. ... This is probably a matter of an external process/script that merges the data spreadsheet download from Central with the source GeoJSON?

Yes, exactly. With this model the merging would have to be customized for your specific form(s) and context. In an entity-based world more of this would (will?) be done by the system. With the extension about accessing properties for use in the form, you could do things like have a shop select1 question and use the property's value (deli in your initial example) as the dynamic default.

@danbjoseph

But otherwise it just shows the border and the translucent fill.

I promise you this is a true story. I looked at the first image you shared and it made perfect sense. Then I stared at the second one for a while and I couldn't tell why you were sharing it. It was only when I came back to your post the second time that I noticed that there are 7 rectangles that are features. Weirdly, this illustrates one reason why I think it's worth having the vertices -- it allows us to have one more visual cue for features. There are so many different basemaps that we won't always get good contrast. Another reason is that it's just simpler implementation-wise to have a single representation. Is there a reason the vertices would be a problem?

@mathieubossaert

Was it my birthday yesterday ?...

Wait, I can't tell whether this is a joke or not!! If not, happy birthday! Either way, I'm glad we were able to get you a suitable gift!

[mathieubossaert]

@LN

Wait, I can't tell whether this is a joke or not!!

It was not a joke

@LN

In an entity-based world more of this would (will?) be done by the system.

Out of the topic but do you mean another way to update media on phones than publishing a whole new version ?

[Ivangayton]

Very excited that this is moving forward! I can't speak unilaterally for HOT, but I do know that we (HOT) have a strategic interest in supporting exactly this: I think at the leastsometime next year we could supply in-kind (dev) support, and maybe more. Shall we book a call?

[LN]

@mathieubossaert

It was not a joke

Amazing!!

@mathieubossaert

do you mean another way to update media on phones than publishing a whole new version ?

Yes, the idea with an entity-based model is that entities would be synced between Collect and the server outside of the context of a single form. So you could have 10k entities and if only one changes, only that one would be updated. Forms would be independent of the entities they act on and wouldn't need to be updated at all in case of entity changes. Hopefully that's where we're going. As you can imagine, it's quite a different model. For now what we've described here would still be the existing model of data updates being tied to forms.

@Ivangayton that's exciting! Will get in touch soon. I think this work as outlined here is ready to move forward but there are lots of related areas where some geo expertise would be very beneficial. In particular, @danbjoseph keeps reminding us that with something like this, offline custom basemaps are even more important.

[LN]

We have made good progress on the implementation here. Given the level of interest, we have taken on several of the extensions described above. In particular, we are making the map appearance work with any select_one variant that specifies a geometry column. That means you could specify a choice list in the choices tab with a geometry column or attach a csv with a geometry column and use those in a select with map appearance. Note that this introduces a new convention: the geometry column is always used to map features when the map appearance is used. For the choices tab or a csv, the column value would be points in ODK format (lat lon alt acc) and eventually also lines and shapes in ODK format.

Because of the way we built that generalization, properties defined in the geojson file will be accessible in expressions the same way that values from other datasets are (I gave a useful example above).

In parallel, @Lindsay_Stevens_Au has added the ability to specify the property to use as the label in the parameters column of the XLSForm (e.g. label=the_label) and that just needs to be reviewed (PR).

Given what we've learned during our implementation, I propose one additional change: in the case of a select from an external geojson file, defaulting to the title property being used as the label. I think we'll want to support styling with the simple style spec sooner than later and so it makes sense to follow it minimally to start with (as opposed to using a name property as I'd previously described).

I look forward to any comments on the proposed expansions and changes I've described above. We still have to wire things together and do quality assurance. We hope to have a beta available by the end of the month.

@LN

Will get in touch soon.

Had a good conversation with @Ivangayton and @danbjoseph with some ideas on where we could use some geo-related help. In particular, they will look into offline vector tiles.

[Florian_May]

Nice! Is that a possible feature for Build?

Support select from map

[ahblake]

@LN

Given the level of interest, we have taken on several of the extensions described above. In particular, we are making the map appearance work with any select_one variant that specifies a geometry column. That means you could specify a choice list in the choices tab with a geometry column or attach a csv with a geometry column and use those in a select with map appearance.

This is extremely exciting news! I can immediately think of many uses for this.

[LN]

No map appearance yet but you can now try your geojson files in v2022.2 Beta 1 to check that we are parsing them correctly. In particular, it should be robust to extra elements.

Any geojson file with .geojson extension, a top-level FeatureCollection and points only should work. The points should have an id property (or top-level id) and a title property. In an XLSForm, add a question of select_one_from_file my-file.geojson. You'll need to use https://getodk.org/xlsform/ or Central v1.4.2+ to convert it. You should see the select in ODK Collect and it should look like a normal select. You should be able to access the geometry and any other property in the usual way. For example, instance("my-file")/root/item[id=${selected}]/geometry

See a small example: geojson.zip (8.3 KB)

[LN]

Beta 2 is rolling out and includes support for the map and map quick appearance for select_one and select_one_from_file! See the post above for a description of how to try it out. There's a sample form with geojson above. This form is an example of an a select with map appearance and a set of locations defined directly in the form.

One thing we are interested in feedback on is performance. If anything feels slow to you, please let us know more about your scenario, the choices you want to load and the basemap you want to use.

More on the beta here. We are currently planning to release the week of August 11th.

[LN]

Thanks to those who have tried out the beta! I want to draw your attention to some test forms in the beta thread which provide ideas on how to use this feature.

[LN]

The initial version of this feature is now available in ODK Collect v2022.2.0. We now have documentation for the select from map question type and for attaching a GeoJSON file to a form.

There are various examples in this post.

Some folks I've talked to have mentioned that the biggest piece missing is status information on device (e.g. I've already visited these places). We hear you! There's no perfect way to do this yet.

This form (+ geojson file) shows two techniques that get pretty close. @Megan_Schroeder @Matt_Berg @Ukang_a_Dickson @Ivangayton

The first involves pulling out the geometry so that the submission will be visible on the form map. The form map only looks for geopoint questions that are user-visible so I couldn't just use a calculate. Instead I put the selection and the pulling of the location in a field-list. If you want to pull out the geometry for other purposes you may be able to just use a calculate.

The second technique that form shows is filtering out locations that have already been visited on the device using last saved. It has limitations. The state isn't preserved across updates of the form definition. There's no way to "un-visit" a place. But it's pretty close and might work well for some scenarios. This is the same technique discussed in this thread but across submissions rather than repeats.

Please try these new features out and let us know how you're using them and what issues you're running into.

Our tentative next step in this general feature area will be to add support for traces and shapes.

[ahblake]

@LN

Please try these new features out and let us know how you're using them and what issues you're running into.

Finally have been able to come up for air and get around to implementing this new feature (only as internal choices so far, GeoJSON is next)... Works a treat! Thanks to everyone involved!

I'm using location and elevation (I have multiple elevation tilesets for my locations) choice filters to prune the location set down to only those that are at that location-elevation combination

When you select a location and it shows the associated fields, is there a logic to what is displayed and the order? When I use internal choices, it shows the label as the title, and then the rest of the fields below the save icon, but not in column L-R or alphabetical A-Z order. Also doesn't appear to be datatype or length.

Ideally I'd be able to force the order / select which are shown, but I think the correct answer is to only populate name label geometry plus my desired fields to be shown and then use the selected name as the index to lookup anything else I want to show / use as dynamic defaults etc.

Trace and shape support would be really really good, as I could skip creating/calculating a centroid for these and show them as is on the map select. And any ability to colour the pins would be super useful as an enumerator guide for different types/ranks.

[LN]

Thanks so much for coming back to give feedback, @ahblake! We're thrilled to hear this is helpful to you.

@ahblake

When you select a location and it shows the associated fields, is there a logic to what is displayed and the order?

If you use GeoJSON or an external CSV, they appear in the order that they have in the source file. For internal choices, as far as I know, pyxform orders them alphabetically by node name and it shouldn't: https://github.com/XLSForm/pyxform/issues/602. If you have a case where the order seems entirely unpredictable that might be something to consider sending us so we can look at it.

@ahblake

only populate name label geometry plus my desired fields to be shown and then use the selected name as the index to lookup anything else

That is a fine solution. It may be less performant and if you do hit a scale where you're running into performance issues, please let us know the item count and the device you're using.

@ahblake

Trace and shape support would be really really good... And any ability to colour the pins

Thanks for briefly describing why you want these, too.

[ahblake]

@LN

Thanks so much for coming back

Have been distracted with non ODK fun things recently, but back at it now!

My current use case is to manually apply point values to existing known anomalies to help guide the enumerator to their location, select it to see details/confirm choice and then update the condition.

Later in the form updated/new anomalies are capturing shape &/or line &/or point information, so in the next cycle, being able to show those in their captured form will be better than a single point.

@LN

For internal choices, as far as I know, pyxform orders them alphabetically by node name and it shouldn't

aha! I see now, I had some mixed capitalisation (labels and acronyms), so it was sorted, but it was A-Za-z - confirmed by adding some test labels just then.

Good to know I can use the order in GeoJSON/CSV or just force the order with creative column titles until the bug is fixed.

The 'jump to extent of points' button is also really useful to zoom to the view of all filtered points.

[ahblake]

@LN

what issues you're running into.

Something I thought of and only just tested; If points have identical coordinates, then only one (the first in the choice list) of those is able to be selected in the Collect map appearance, but all are visible for selection in the select_one Enketo appearance.

What I haven't yet tested is how close items can be before they become indistinguishable for selection at the max zoom levels that the different base layers allow (I only use mapbox as the others are too limited), i.e. is this actually a problem that requires solving - are multiple selections with identical/near identical coordinates a valid use case?

The expected behaviour I'd imagine is some sort of clustering where you select the cluster and it expands to show all points at those coordinates like this:

from here