Ability to load locations into the geowidget to guide data collection

Here is the library I referred to. It's pretty robust and is being used in OpenSRP, DHIS2 tracker and I believe CommCare soon.

I've been chatting with @Ivangayton / HOT OSM about how to resume some of the past work around improving the geo widgets that @zestyping was such a key part of, and also develop the ideas for the work into a more cohesive vision. There were community discussions around this at the most recent ODK Convening in late 2019, and a variety of forum discussions. We're just wrapping up on an initial document to describe the aspirations and so it has not been discussed with the TSC yet. The hope was to get additional eyes on the ideas and work with the TSC to figure out the technical feasibility and fit with the ODK project. Then if we can agree on a clear scope and plans, move on to identification of resources to make happen.

EDIT: I think there is a lot of overlap between the above and our proposed roadmap ideas.

2 Likes

As @danbjoseph alludes to, the general space of selecting a geo feature as part of the data collection process is now being taken up more actively by the TSC. The geo future directions doc gives a good overview of the various themes that users have brought up related to all things geo and that the dev team would like to take up. The map-centered form workflow section in that doc is closely related to what is being described here and what @danbjoseph has started fleshing out in the doc linked above.

@Matt_Berg, could you please share a writeup of the specific project/workflow that is triggering this request? A good example of such a writeup is @J_D1’s View GPS point on Map for Blank Forms. I think the biggest decision to make is whether to treat this as a subset of entity-based data collection or not and getting a clear understanding of what workflows need to be supported would really help. Specifically, the first post in this thread suggests attaching the geo features data to a single form. @danbjoseph describes a model where the feature selection happens before form selection (e.g. select a waterpoint and then select what form to fill about it) which means features data has to come from outside the context of a single form.

There are excellent notes from the last TSC call here. @Ukang_a_Dickson brought up various related projects and I imagine you’ve probably synced up (I mention this because @Matt_Berg and @Ukang_a_Dickson are both from Ona).

Since there are a lot of moving pieces in this general space, there would definitely need to be detailed user stories leading to a specification and iteration with the TSC. @Matt_Berg, what is your timeline and would you be up for working towards a specification for the TSC to consider?

Regarding Kujaku, the systems that incorporate it are all entity-centric and I assume that those systems use it as a way to visualize and collect data about those entities. Without an entity context, that pushes more data/workflow management onto users. @Matt_Berg do systems that incorporate the library build their Kujaku configs on the client or server? Regardless, leveraging the library might be possible. An immediate question that would need to be explored is how the existing Collect point/trace/shape question types that use the Mapbox API coexist with the Kujaku lib. Kujaku pulls various functionality from Collect like offline mbtiles management, location provider management and both rely on the same underlying Mapbox lib but Kujaku has an older version. I think that almost certainly means conflicts.

2 Likes

Hi @LN,

Do you know if there is an update on this?

Thanks,
JD

The Technical Advisory Board has assembled a working group exploring entity-centered workflows. It's likely that this would include this kind of mapping. The group is reviewing use cases and process for making bigger shifts to the tools as well as possible targeted funding opportunities.

3 Likes

The Entity-Based Data Collection Working Group was formed, met, came up with a high-level project plan, and is now seeking funding. Geo will eventually be central to what we have in mind but is currently looking like it will be layered on top of other upfront work. In the meantime, Ona has expressed renewed interest in what @Matt_Berg described above.

We propose adding a map appearance and geojson support to select_one_from_file. Initially the map appearance would only work with geojson files but we could expand it to work with all select_one variants (see extensions below). select_one_from_file with a geojson file and without the map appearance would provide a text interface identical to how select_one_from_file works today.

The user experience provided by the map appearance would be similar to a geopoint with the placement-map appearance. In the form, question text and a button with text “Select place” would be displayed. If a feature has previously been selected, its label (from the name property, see below) would be displayed.

When the “Select place” button is tapped, a map view would be launched. All features from the geojson file would be displayed on top of the basemap and any reference layer configured on the device. Instead of being able to select an arbitrary location, the user would be able to tap on any feature defined in the geojson file.

geojson files used with select_one_from_file:

  • MUST have a .geojson extension (not .json)
  • MUST define a single top-level FeatureCollection
  • MUST define a property called id for each feature in the collection (features without this property will be silently ignored; see extensions below for more flexibility which could be later introduced)
  • MAY define a property called name for some or all features in the collection which will be used as the feature’s label (Note: I propose name because I’ve seen that in several geojson file examples. An alternative would be title which is specified by simplestyle. interested in opinions here!)
  • MAY define any number of additional custom properties for some or all features which will be displayed when a feature is selected (and eventually available to form logic, see extensions)

Points would be displayed as markers like the ones used for the filled form map . Additional feature types could be supported as described in the extensions below.

All of the features would be of the primary app color (blue-green) when unselected. The feature selected would be displayed in red to match the existing geo widgets. In the case of points, the marker would also increase in size (same as the filled form map).

When the quick appearance is specified in addition to the map appearance, tapping on a feature would select it immediately and the map screen would close.

When the quick appearance is not specified and a feature is tapped, a bottom sheet would be shown as on the filled form map. If the feature has a property called name, that property’s value will be shown as its label (above the horizontal rule). If there is no such property, the id/underlying value will be displayed as the label. Any properties in addition to id and name will be displayed in a table below the label.

When a feature is selected and the save icon is tapped, that feature’s id/underlying value would be saved as the question’s value, overwriting any previously-saved value (same as geotrace/shape). When the trashcan icon is tapped, the current selection is abandoned and any previously-saved value remains (same as geotrace/shape).

When a geojson file is used without the map appearance, the id property would be used as the underlying value/name and the name property would be used as the display value/label.

An additional requirement is the possibility to specify a feature not defined in the geojson file. For now, we propose pushing that to form design: the form definition could include label text that prompts data collectors to leave the question blank if the feature isn’t on the map. If the value is blank, a series of questions could be asked to define the new feature. Once we have the basic requirements in place, we’ll try this and see if it’s sufficiently clear.

Extensions:

  • Add feature types
    • MultiPoint, LineString, MultiLineString as outlined circles with a white center separated by lines as for the geotrace widget
    • Polygon, MultiPolygon as outlined circles with a white center separated by lines as for the geoshape widget with the addition of a light shading of the enclosed area
  • Any property defined in the geojson file could be accessed in expressions the same way that values from other datasets are. For example, if a field called the_road has type select_one_from_file roads.geojson in an XLSForm, the repair_state property of the road selected could be accessed with expression instance('roads')/root/item[id=${the_road}]/repair_state. This would make it possible to do things like constrain the features that can be selected based on the current location or filter the features shown based on some condition (e.g. only show hospitals that have some specialized team)
  • Add flexibility to the geojson requirements:
    • if there is no property called id, look for an id foreign member (sibling of type: https://datatracker.ietf.org/doc/html/rfc7946#section-6.1)
    • allow specifying the property to use as the underlying value in the parameters column of the XLSForm (e.g. name=the_id)
    • allow specifying the property to use as the label in the parameters column of the XLSForm (e.g. label=the_label)
  • Support map appearance on select_one. This would also require specifying either lat and lon columns or a geojson column from the parameters column of the XLSForm. (XForms spec might be weird; additional children for itemset? This is why we propose introducing the map appearance rather than only using the filename extension)
  • Support map appearance on select_multiple_from_file/select_multiple
  • Allow styling using the simple style spec. This would make it possible for the geojson file to specify things like color for individual features.
6 Likes

I'm super excited to read this. Two initial trains of thought. I'm sure there will be more. Please excuse the rambling as I work through my thoughts.


[1] I'm trying to think through how I would get survey answers back into the geospatial dataset. That is, if I have a FeatureCollection of buildings. And they all have some assortment of the additional custom properties (outside of the must have id). If a feature has building=yes + shop=deli and then my survey data is that actually shop=cheese + also levels=3 then how do I get my output FeatureCollection that has that feature with all three of the right tags building=yes; shop=cheese; levels=3. This is probably a matter of an external process/script that merges the data spreadsheet download from Central with the source GeoJSON?

The scenario where we may only be adding details, like a damage assessment where we are adding a damage_level=* custom property that no feature has yet, seems pretty simple in then getting back to your desired output for analysis (basic join of a survey data table to the FeatureCollection in a GIS software on the id field).

A mapping scenario in which we might be adding/ editing/ removing custom properties, seems more complicated. If someone clicks a feature with amenity=place_of_worship then we have a questions asking them if they can add or update religion=* and denomination=*. If a feature has shop=* then we ask if they can add or update name=*. And so on... and then we want to bring together the starting FeatureCollection data with the assortment of new/ changed/ removed attributes.


[2] When would a user need to see the vertices of a polygon/multipolygon? The geoshape widget in ODK is for editing the shape and so you need to know where the points are. In the OSM iD editor it shows the points when a shape is selected for editing. But otherwise it just shows the border and the translucent fill.
Screen Shot 2021-12-01 at 5.08.42 PM

2 Likes

Was it my birthday yesterday ?...
Yes it really was :star_struck:

4 Likes

I believe this would enable using either a choice filter to filter out features based on a previously-captured location, right? So you could require someone capture their location, then once they do that, they only see features that are say 20 meters of their location?

Edit: Yes because I see now that you also said...

1 Like

Yes, exactly. With this model the merging would have to be customized for your specific form(s) and context. In an entity-based world more of this would (will?) be done by the system. With the extension about accessing properties for use in the form, you could do things like have a shop select1 question and use the property's value (deli in your initial example) as the dynamic default.

I promise you this is a true story. I looked at the first image you shared and it made perfect sense. Then I stared at the second one for a while and I couldn't tell why you were sharing it. It was only when I came back to your post the second time that I noticed that there are 7 rectangles that are features. Weirdly, this illustrates one reason why I think it's worth having the vertices -- it allows us to have one more visual cue for features. There are so many different basemaps that we won't always get good contrast. Another reason is that it's just simpler implementation-wise to have a single representation. Is there a reason the vertices would be a problem?

Wait, I can't tell whether this is a joke or not!! If not, happy birthday! Either way, I'm glad we were able to get you a suitable gift! :tada: :smiling_face_with_three_hearts:

1 Like

It was not a joke :joy::sob:

Out of the topic but do you mean another way to update media on phones than publishing a whole new version ?

1 Like

Very excited that this is moving forward! I can't speak unilaterally for HOT, but I do know that we (HOT) have a strategic interest in supporting exactly this: I think at the leastsometime next year we could supply in-kind (dev) support, and maybe more. Shall we book a call?

3 Likes

Amazing!! :cake:

Yes, the idea with an entity-based model is that entities would be synced between Collect and the server outside of the context of a single form. So you could have 10k entities and if only one changes, only that one would be updated. Forms would be independent of the entities they act on and wouldn't need to be updated at all in case of entity changes. Hopefully that's where we're going. As you can imagine, it's quite a different model. For now what we've described here would still be the existing model of data updates being tied to forms.

@Ivangayton that's exciting! Will get in touch soon. I think this work as outlined here is ready to move forward but there are lots of related areas where some geo expertise would be very beneficial. In particular, @danbjoseph keeps reminding us that with something like this, offline custom basemaps are even more important.

2 Likes

5 posts were split to a new topic: When are media files re-downloaded?

We have made good progress on the implementation here. Given the level of interest, we have taken on several of the extensions described above. In particular, we are making the map appearance work with any select_one variant that specifies a geometry column. That means you could specify a choice list in the choices tab with a geometry column or attach a csv with a geometry column and use those in a select with map appearance. Note that this introduces a new convention: the geometry column is always used to map features when the map appearance is used. For the choices tab or a csv, the column value would be points in ODK format (lat lon alt acc) and eventually also lines and shapes in ODK format.

Because of the way we built that generalization, properties defined in the geojson file will be accessible in expressions the same way that values from other datasets are (I gave a useful example above).

In parallel, @Lindsay_Stevens_Au has added the ability to specify the property to use as the label in the parameters column of the XLSForm (e.g. label=the_label) and that just needs to be reviewed (PR).

Given what we've learned during our implementation, I propose one additional change: in the case of a select from an external geojson file, defaulting to the title property being used as the label. I think we'll want to support styling with the simple style spec sooner than later and so it makes sense to follow it minimally to start with (as opposed to using a name property as I'd previously described).

I look forward to any comments on the proposed expansions and changes I've described above. We still have to wire things together and do quality assurance. We hope to have a beta available by the end of the month.

Had a good conversation with @Ivangayton and @danbjoseph with some ideas on where we could use some geo-related help. In particular, they will look into offline vector tiles.

5 Likes

Nice! Is that a possible feature for Build?

This is extremely exciting news! I can immediately think of many uses for this.

1 Like

No map appearance yet but you can now try your geojson files in v2022.2 Beta 1 to check that we are parsing them correctly. In particular, it should be robust to extra elements.

Any geojson file with .geojson extension, a top-level FeatureCollection and points only should work. The points should have an id property (or top-level id) and a title property. In an XLSForm, add a question of select_one_from_file my-file.geojson. You'll need to use https://getodk.org/xlsform/ or Central v1.4.2+ to convert it. You should see the select in ODK Collect and it should look like a normal select. You should be able to access the geometry and any other property in the usual way. For example, instance("my-file")/root/item[id=${selected}]/geometry

See a small example: geojson.zip (8.3 KB)

1 Like

Beta 2 is rolling out and includes support for the map and map quick appearance for select_one and select_one_from_file! See the post above for a description of how to try it out. There's a sample form with geojson above. This form is an example of an a select with map appearance and a set of locations defined directly in the form.

One thing we are interested in feedback on is performance. If anything feels slow to you, please let us know more about your scenario, the choices you want to load and the basemap you want to use.

More on the beta here. We are currently planning to release the week of August 11th.

2 Likes

Thanks to those who have tried out the beta! I want to draw your attention to some test forms in the beta thread which provide ideas on how to use this feature.