Ability to load locations into the geowidget to guide data collection

1. What is the general goal of the feature?

The goal of the feature is to make it possible to load in a series of locations (eg. health facilities) into ODK and make them viewable on a map similar to the way you can currently view submissions now (see ticket below). Clicking on the point would allow you to pass metadata (eg. the feature id) to the submission. This would provide similar "check-in" functionality that OMK provides.

More feature specifics

  • The source for the locations would be a CSV or geojson file downloaded as a form media file
  • In the form ability to launch the geowidget view, view nearby locations, click on them and have that populate the form with specified properties from the locations file
  • In the form definition, define the properties to pull from the location file when you click on it.

2. What are some example use cases for this feature?

Anyone wanting to visit specific locations to collect data. This is particularly the case where you are collecting data on things (eg. waterpoints) or locations (hamlets) that don't have clear names.

3. What can you contribute to making this feature a reality?

We would be up for helping to develop it. We have already developed a different Mapbox based Geowidget capable of providing a lot of this functionality. Part of our analysis is to figure out if we should develop a solution based on that would pair our map app with ODK (similar to OMK) to achieve this. If ODK's map widget has the needed based functionality we would be open to building out the functionality here.

Wanted to start this conversation to understand if #1 people thought this would be useful and #2 how viable using ODK's map widget as a starting point would be for this.

At first glance, this looks like some functionality that is on the general roadmap. I think the biggest challenge is coming up with specifications that are coherent with what already exists and reasonable for tools across the ecosystem to support.

What's higher priority, this or Collect will need to stop using /sdcard/odk for files - #23 by Kigamba? Could y'all maybe free up some cycles to think through this by doing pyxform/XLSForm review?

Hi @LN they both are important and we can potentially invest time in both so I would rather keep them separate when we try and figure out the approach for this.

If you have a bit more clarity on the roadmap that would be helpful. We can spend more time on how this would be represented in XLSForm spec if that is what you are requesting.

At a high level, I think we would probably want to be able to define in the geowidget question type the file being referenced and the metadata that would be passed back. We can probably assume for now that the locations file only have one set of GPS points per feature to start.

We've got a few big things on our plate at the moment so it would help to know the relative priority since both will need cycles from the core ODK team. I'll try to get some links to prior conversations soon.

Yes, the first step is defining the scope, then it's coming up with an XForms proposal, then an XLSForm proposal.

2 Likes

Cool thank you that is very helpful to understand. Will speak internally re: XLSForms and get back.

1 Like

CC @danbjoseph @Ivangayton @zestyping @russbiggs_hotosm who have been involved in related prior conversations.

Here is the library I referred to. It's pretty robust and is being used in OpenSRP, DHIS2 tracker and I believe CommCare soon.

I've been chatting with @Ivangayton / HOT OSM about how to resume some of the past work around improving the geo widgets that @zestyping was such a key part of, and also develop the ideas for the work into a more cohesive vision. There were community discussions around this at the most recent ODK Convening in late 2019, and a variety of forum discussions. We're just wrapping up on an initial document to describe the aspirations and so it has not been discussed with the TSC yet. The hope was to get additional eyes on the ideas and work with the TSC to figure out the technical feasibility and fit with the ODK project. Then if we can agree on a clear scope and plans, move on to identification of resources to make happen.

EDIT: I think there is a lot of overlap between the above and our proposed roadmap ideas.

2 Likes

As @danbjoseph alludes to, the general space of selecting a geo feature as part of the data collection process is now being taken up more actively by the TSC. The geo future directions doc gives a good overview of the various themes that users have brought up related to all things geo and that the dev team would like to take up. The map-centered form workflow section in that doc is closely related to what is being described here and what @danbjoseph has started fleshing out in the doc linked above.

@Matt_Berg, could you please share a writeup of the specific project/workflow that is triggering this request? A good example of such a writeup is @J_D1’s View GPS point on Map for Blank Forms - #10 by J_D1. I think the biggest decision to make is whether to treat this as a subset of entity-based data collection or not and getting a clear understanding of what workflows need to be supported would really help. Specifically, the first post in this thread suggests attaching the geo features data to a single form. @danbjoseph describes a model where the feature selection happens before form selection (e.g. select a waterpoint and then select what form to fill about it) which means features data has to come from outside the context of a single form.

There are excellent notes from the last TSC call here. @Ukang_a_Dickson brought up various related projects and I imagine you’ve probably synced up (I mention this because @Matt_Berg and @Ukang_a_Dickson are both from Ona).

Since there are a lot of moving pieces in this general space, there would definitely need to be detailed user stories leading to a specification and iteration with the TSC. @Matt_Berg, what is your timeline and would you be up for working towards a specification for the TSC to consider?

Regarding Kujaku, the systems that incorporate it are all entity-centric and I assume that those systems use it as a way to visualize and collect data about those entities. Without an entity context, that pushes more data/workflow management onto users. @Matt_Berg do systems that incorporate the library build their Kujaku configs on the client or server? Regardless, leveraging the library might be possible. An immediate question that would need to be explored is how the existing Collect point/trace/shape question types that use the Mapbox API coexist with the Kujaku lib. Kujaku pulls various functionality from Collect like offline mbtiles management, location provider management and both rely on the same underlying Mapbox lib but Kujaku has an older version. I think that almost certainly means conflicts.

2 Likes

Hi @LN,

Do you know if there is an update on this?

Thanks,
JD

The Technical Advisory Board has assembled a working group exploring entity-centered workflows. It's likely that this would include this kind of mapping. The group is reviewing use cases and process for making bigger shifts to the tools as well as possible targeted funding opportunities.

3 Likes

The Entity-Based Data Collection Working Group was formed, met, came up with a high-level project plan, and is now seeking funding. Geo will eventually be central to what we have in mind but is currently looking like it will be layered on top of other upfront work. In the meantime, Ona has expressed renewed interest in what @Matt_Berg described above.

We propose adding a map appearance and geojson support to select_one_from_file. Initially the map appearance would only work with geojson files but we could expand it to work with all select_one variants (see extensions below). select_one_from_file with a geojson file and without the map appearance would provide a text interface identical to how select_one_from_file works today.

The user experience provided by the map appearance would be similar to a geopoint with the placement-map appearance. In the form, question text and a button with text “Select place” would be displayed. If a feature has previously been selected, its label (from the name property, see below) would be displayed.

When the “Select place” button is tapped, a map view would be launched. All features from the geojson file would be displayed on top of the basemap and any reference layer configured on the device. Instead of being able to select an arbitrary location, the user would be able to tap on any feature defined in the geojson file.

geojson files used with select_one_from_file:

  • MUST have a .geojson extension (not .json)
  • MUST define a single top-level FeatureCollection
  • MUST define a property called id for each feature in the collection (features without this property will be silently ignored; see extensions below for more flexibility which could be later introduced)
  • MAY define a property called name for some or all features in the collection which will be used as the feature’s label (Note: I propose name because I’ve seen that in several geojson file examples. An alternative would be title which is specified by simplestyle. interested in opinions here!)
  • MAY define any number of additional custom properties for some or all features which will be displayed when a feature is selected (and eventually available to form logic, see extensions)

Points would be displayed as markers like the ones used for the filled form map . Additional feature types could be supported as described in the extensions below.

All of the features would be of the primary app color (blue-green) when unselected. The feature selected would be displayed in red to match the existing geo widgets. In the case of points, the marker would also increase in size (same as the filled form map).

When the quick appearance is specified in addition to the map appearance, tapping on a feature would select it immediately and the map screen would close.

When the quick appearance is not specified and a feature is tapped, a bottom sheet would be shown as on the filled form map. If the feature has a property called name, that property’s value will be shown as its label (above the horizontal rule). If there is no such property, the id/underlying value will be displayed as the label. Any properties in addition to id and name will be displayed in a table below the label.

When a feature is selected and the save icon is tapped, that feature’s id/underlying value would be saved as the question’s value, overwriting any previously-saved value (same as geotrace/shape). When the trashcan icon is tapped, the current selection is abandoned and any previously-saved value remains (same as geotrace/shape).

When a geojson file is used without the map appearance, the id property would be used as the underlying value/name and the name property would be used as the display value/label.

An additional requirement is the possibility to specify a feature not defined in the geojson file. For now, we propose pushing that to form design: the form definition could include label text that prompts data collectors to leave the question blank if the feature isn’t on the map. If the value is blank, a series of questions could be asked to define the new feature. Once we have the basic requirements in place, we’ll try this and see if it’s sufficiently clear.

Extensions:

  • Add feature types
    • MultiPoint, LineString, MultiLineString as outlined circles with a white center separated by lines as for the geotrace widget
    • Polygon, MultiPolygon as outlined circles with a white center separated by lines as for the geoshape widget with the addition of a light shading of the enclosed area
  • Any property defined in the geojson file could be accessed in expressions the same way that values from other datasets are. For example, if a field called the_road has type select_one_from_file roads.geojson in an XLSForm, the repair_state property of the road selected could be accessed with expression instance('roads')/root/item[id=${the_road}]/repair_state. This would make it possible to do things like constrain the features that can be selected based on the current location or filter the features shown based on some condition (e.g. only show hospitals that have some specialized team)
  • Add flexibility to the geojson requirements:
    • if there is no property called id, look for an id foreign member (sibling of type: https://datatracker.ietf.org/doc/html/rfc7946#section-6.1)
    • allow specifying the property to use as the underlying value in the parameters column of the XLSForm (e.g. name=the_id)
    • allow specifying the property to use as the label in the parameters column of the XLSForm (e.g. label=the_label)
  • Support map appearance on select_one. This would also require specifying either lat and lon columns or a geojson column from the parameters column of the XLSForm. (XForms spec might be weird; additional children for itemset? This is why we propose introducing the map appearance rather than only using the filename extension)
  • Support map appearance on select_multiple_from_file/select_multiple
  • Allow styling using the simple style spec. This would make it possible for the geojson file to specify things like color for individual features.
6 Likes

I'm super excited to read this. Two initial trains of thought. I'm sure there will be more. Please excuse the rambling as I work through my thoughts.


[1] I'm trying to think through how I would get survey answers back into the geospatial dataset. That is, if I have a FeatureCollection of buildings. And they all have some assortment of the additional custom properties (outside of the must have id). If a feature has building=yes + shop=deli and then my survey data is that actually shop=cheese + also levels=3 then how do I get my output FeatureCollection that has that feature with all three of the right tags building=yes; shop=cheese; levels=3. This is probably a matter of an external process/script that merges the data spreadsheet download from Central with the source GeoJSON?

The scenario where we may only be adding details, like a damage assessment where we are adding a damage_level=* custom property that no feature has yet, seems pretty simple in then getting back to your desired output for analysis (basic join of a survey data table to the FeatureCollection in a GIS software on the id field).

A mapping scenario in which we might be adding/ editing/ removing custom properties, seems more complicated. If someone clicks a feature with amenity=place_of_worship then we have a questions asking them if they can add or update religion=* and denomination=*. If a feature has shop=* then we ask if they can add or update name=*. And so on... and then we want to bring together the starting FeatureCollection data with the assortment of new/ changed/ removed attributes.


[2] When would a user need to see the vertices of a polygon/multipolygon? The geoshape widget in ODK is for editing the shape and so you need to know where the points are. In the OSM iD editor it shows the points when a shape is selected for editing. But otherwise it just shows the border and the translucent fill.
Screen Shot 2021-12-01 at 5.08.42 PM

2 Likes

Was it my birthday yesterday ?...
Yes it really was :star_struck:

4 Likes

I believe this would enable using either a choice filter to filter out features based on a previously-captured location, right? So you could require someone capture their location, then once they do that, they only see features that are say 20 meters of their location?

Edit: Yes because I see now that you also said...

1 Like

Yes, exactly. With this model the merging would have to be customized for your specific form(s) and context. In an entity-based world more of this would (will?) be done by the system. With the extension about accessing properties for use in the form, you could do things like have a shop select1 question and use the property's value (deli in your initial example) as the dynamic default.

I promise you this is a true story. I looked at the first image you shared and it made perfect sense. Then I stared at the second one for a while and I couldn't tell why you were sharing it. It was only when I came back to your post the second time that I noticed that there are 7 rectangles that are features. Weirdly, this illustrates one reason why I think it's worth having the vertices -- it allows us to have one more visual cue for features. There are so many different basemaps that we won't always get good contrast. Another reason is that it's just simpler implementation-wise to have a single representation. Is there a reason the vertices would be a problem?

Wait, I can't tell whether this is a joke or not!! If not, happy birthday! Either way, I'm glad we were able to get you a suitable gift! :tada: :smiling_face_with_three_hearts:

1 Like

It was not a joke :joy::sob:

Out of the topic but do you mean another way to update media on phones than publishing a whole new version ?

1 Like

Very excited that this is moving forward! I can't speak unilaterally for HOT, but I do know that we (HOT) have a strategic interest in supporting exactly this: I think at the leastsometime next year we could supply in-kind (dev) support, and maybe more. Shall we book a call?

3 Likes

Amazing!! :cake:

Yes, the idea with an entity-based model is that entities would be synced between Collect and the server outside of the context of a single form. So you could have 10k entities and if only one changes, only that one would be updated. Forms would be independent of the entities they act on and wouldn't need to be updated at all in case of entity changes. Hopefully that's where we're going. As you can imagine, it's quite a different model. For now what we've described here would still be the existing model of data updates being tied to forms.

@Ivangayton that's exciting! Will get in touch soon. I think this work as outlined here is ready to move forward but there are lots of related areas where some geo expertise would be very beneficial. In particular, @danbjoseph keeps reminding us that with something like this, offline custom basemaps are even more important.

2 Likes

5 posts were split to a new topic: When are media files re-downloaded?