ODK Collect v2023.1 Beta: select traces and shapes from map, Google Drive deprecation banner

Releases Betas
1

ODK Collect betas are an opportunity to get community feedback on upcoming releases. If you have an ongoing data collection campaign, we recommend quickly verifying your form on a test device. The release will be delayed until all reported issues are fixed.

The release is currently planned for mid-March. We will periodically release betas to get feedback on new features and fixes.

Joining the beta program
To join the beta program, find ODK Collect in the Play Store on your device (not in the web browser) and scroll all the way down. Please don't join the beta with a device or account actively used for data collection! In particular, note that joining the beta is account-based. If you use the same Google account across multiple devices, do not join the beta with that account.

Leaving the beta program
You can leave the beta program from the bottom of the Play Store at any time. Once you leave, you will get the next production update when it is released. If you need to go back to the previous production release, uninstall and reinstall the app. Your settings will be reset but your forms will remain (though backups are always recommended).

What to check in this release

  • HTTP caching with ETag/If-None-Match. This can limit data transferred in some cases and allows servers to provide dynamic form attachments without Collect constantly redownloading them. It has only been verified with Central. If a server provides incorrect ETag values, form updates may be missed.
  • Select traces and shapes from map. You should now be able to include traces and shapes in your geoJSON, CSV or internal datasets for use with the map appearance on select one. We're not yet happy with the styling of those traces and shapes, especially when they're selected. We welcome ideas to make it clearer when one is selected.
  • Google Drive deprecation banner displayed for Google Drive projects

Additional testing
You can find a full list of changes in the release notes:

This beta includes a beta-only experimental setting that demonstrates initial support for building entities from filled forms. If you build a form according to the entities specification (you can use this pyxform branch), you will be able to go to Settings > Experimental > Entities and see entities created after you finalized filled forms. These entities will only exist until Collect goes out of memory. These experimental settings will not be included in the production build.

Thanks to all testers for your help!

1 Like
2

I love it. :heavy_heart_exclamation:

If the shape extends off screen due to zoom level, selecting it doesn't adjust zoom to extents of selected shape. It does appear to centre the first (?) vertice on screen? Could it trigger a similar action to the existing zoom-extent map button but for the selected element?

Selecting a shape is only possible by tapping a vertice, not on an edge or interior, this is ok but my first reaction was to tap inside the shape, I'd add user guidance for this as a hint in my forms.

Close / overlapping shapes are discernable as there's no shading, and selection is easy enough by tapping a vertice even if it's "inside" another shape. (Very similar shapes would have similar issues selecting them to very close points as previously mentioned)

With Google-streets basemap with/without a tileset, when the shape overlapped with a building element, the edges/lines were greyed out as if the building was partially transparent and above them, but below the vertices. Seems to only affect this particular basemap.
image
image

With raster tilesets in other basemaps, the lines appear over the top of the tileset as in geoshape widget
image

Mixed formats - All three can be represented at once, doesn't matter what's in the geometry column. I hadn't expected this, but it's great.

image
image1282×1602 77.1 KB

Error handling - I opened my updated form (CSV had geometry values as polygons not points) in an older non beta version, and they displayed as points. I didn't check which point (first/centroid etc), but this was a graceful fallback.

There is no apparent visual change in selected shape - remains as red lines with white filled red circles. A change in colour, line weight, vertice style/fill or perhaps colour and modify line style from solid to dashed for colourblind people would help.

The top shape is selected here:

image
image1164×586 38.6 KB

Running on an old Galaxy S5, so it might not be representative: Mapbox basemap is horribly slow and result in ODK Collect isn't responding notices. Google basemap is much snappier on the same device. 31 rectangles in the test select. I haven't compared this to select a point on this device.
Testing the same form on a Galaxy Tab S7 there was no noticeable slowdown, so not a real issue for me, but it did just crash when switching from google to mapbox and opening the select from map question, hasn't repeated yet. Edit happened again when loading select from map, without switching basemap first.

image
image2290×834 122 KB

1 Like
3

@ahblake thanks for testing and sharing your thoughts.

Yes, we know about this bug and will try to address it soon.

Agreed, we need to think about it.

That's interesting... does Mapbox work well in other places like the map of forms or geotrace/geoshape widgets (any place where maps are displayed)

1 Like
4

Hi,

Can you share an example of a geoJSON and a CSV with traces and shapes?

Thanks.

5

This is a form I used for testing and it contains a select one from map question with three different sources:

  • XML
  • csv
  • geojson

SelectOneFromMap.zip (9.6 KB)

6

Ok, I can make it crash on demand now, at least with one particular form.

  • With mapbox base selected
  • Load form
  • Answer a couple of initial questions to activate the geoshape
  • Open geoshape widget, exit without creating a shape
    • There is a pause after selecting exit and the loading form dialogue that usually appears when opening the form appears briefly before being returned to the form
  • Opening either the geoshape or select from map will result in the 'no implementation found' crash error.
  • Selecting OK to dismiss error and then Fill blank form loads the form navigation view and it recovers the values already entered.

It looks like the error is triggered by opening the geoshape widget but not creating a shape and exiting. Using the select from map to pick an existing item preloads a shape to the geoshape, so opening it to view, then editing or leaving as it and exiting or saving and exiting doesn't cause issues.

Exiting geoshape without creating a shape and then exiting the form and discarding results in this alternate crash message

image
image1968×650 60.3 KB

Changing to all other basemaps, the pause then loading dialogue when exiting geoshape without creating one occurs, geoshape/select from map can both then be used and there is no crash, however exiting the form will cause the above alternate crash with the other basemaps.

In 2022.4.4 there is no pause exiting the geoshape after not creating a shape, and no crash on exit or opening geoshape/select from map.

7

@ahblake could you attach your form? I wonder if it makes a difference or maybe you are able to reproduce the crash with any form?

8

I stripped the form down successively and it's not related to select from map at all.
ODKBeta_geo_crash.xlsx (35.9 KB)

A form with just a geotrace/shape that is opened, not populated and exited without saving will pause for a moment before returning to the form, and trigger the crash (on exiting and discarding with all basemaps, or on reloading another geowidget on mapbox). Exiting without populating by saving as blank does not trigger the crash.

geopoint with placement-map creates a point as soon as accuracy is reached, so trashing it and saving the cleared state is like saving blank above so doesn't crash.

Unrelated, I tried to open a form in Enketo that had a CSV of locations plus marker-color and marker-symbol columns, and this failed.

CSV column heading "marker-color" cannot be turned into a valid XML element

removing marker-color then resulted in

CSV column heading "marker-symbol" cannot be turned into a valid XML element

removing marker-symbol then let the form load in Enketo.

9

I am really interested in this function but don't have a spare account / device to join the Beta, so apologies for using 'second-hand' information... hopefully I haven't misinderstood

The ability to select a polygon by tapping within its area would be intuitive and useful:
intuitive because that is how I am accustomed to working with GIS data and a polygon in spatial terms (for me) usually represents a 'solid' area, rather than just a perimeter or series of vertices. and
useful because we often work with agricultural data where field-boundaries are coincident... it would be very hard to reliably select a polygon if it has a shared perimeter with one or more other polygons.

If it is possible to have a semi-transparent fill for polygons and increase or decrease the transparency when selected (or preferably change the outline colour / line-thickness) that would be really useful.

Currently I use a custom raster basemap (mbtiles) with the polygons visible and a point at the centroid that can be selected using the existing functionality. And when dealing with lines (in my case mostly footpaths) I use a point at the first vertex of the line that can be selected (again the line is visible on my custom basemap). And I have been manually using these as entities to pre-fill (parts of) the form.

Moving to polygons and lines 'natively' within the Collect dataset is really exciting (sorry, did I say that out loud?) - it might even help me avoid distributing custom mbtiles layers (which is problematic when devices are locked and the Android folder is inaccessible!)

2 Likes
10

There's a new beta out with a couple of fixes and one significant addition: HTTP caching. This has only been verified with Central so folks who use alternative servers may want to give it a try. The biggest risk is that if a server returns inappropriate ETags some form updates may not happen. It's not a problem if a server returns no ETags. The main benefit of introducing HTTP caching is that it gives servers the option of streaming dynamic responses to form attachment requests without having to pre-compute file hashes.

I believe this was https://github.com/getodk/collect/issues/5453 which is fixed in the latest beta. Thanks!

Another good find, thank you. This has been addressed in https://github.com/enketo/enketo-express/pull/530. We don't have immediate release plans but if it's blocking you, please let us know.

That makes a lot of sense, thanks for explaining it so well. It looks like we won't get this into the current release but we will try to add it soon.

Again, we're going to look at this for a future release. We'll try to get a change in outline in. For now we're going to be zooming the selected shape/trace to fit (not yet in this beta).

Yes, that's one of our goals for this!

2 Likes
11

We now have added polygon selection by tapping within its area! Please take a look at the latest beta and share any feedback you may have. One small change we're considering is to make the fill more transparent and/or make the red less bright/blue.

We expect that this will be the last beta before we release.

1 Like
12

Anxiously awaiting beta 4 to update on my device...sounds promising, will edit when I can test it.

Edit - it's here!

Ok:

  • Using a mixed set of polygon/trace/point, with some polygons overlapping each other
  • Testing on Galaxy Tab S7, Android 13.
  • When selected, the zoom to an extent a bit larger than the element is great.
  • Trace can still only be selected on the vertices (ok for me for now, if you could eventually select via the linear section(s) that would be a nice to have) Traces can be selected by edges or vertices, but requires more accuracy at higher zoom levels.
  • Polygons are shaded transparent red and can be selected by the vertices or anywhere inside. :clap:
  • If polygons overlap, the one 'underneath' (I don't know if this is file order or something else, haven't investigated) cannot be selected by tapping it's area a second time to toggle, but you can select the one underneath via the vertices so it's not blocked from selection.
  • Overlapping polygons add the transparencies, tileset still visible but less clear.
  • Raster tileset is still visible through transparency, perhaps lighten it a bit more to lessen the effect? Changing to blue might be harder to see on maps with areas of water, whereas red is less likely to be used for land/water/etc elements in maps
  • Apart from the zoom effect, it's not totally clear when there are overlapping or closely located traces / polygons which is selected as there is no change in styling etc to indicate this like how a point increases in size when selected.
13

I'm ashamed that I haven't yet found the time to test these features that will be very useful to us :sob:

14

Are you able to double-check on that? I'm able to click on the lines using any of the basemaps (Google, Mapbox etc). If not, could you share the form/media files you're using?

15

Out of interest: do you have real life use cases that involve overlapping polygons?

16

Ah, I see what happened - I was zoomed too close and kept missing the hitbox - have to be pretty centred on the line that close, took me a few tries at the highest zoom levels to get it. Zooming out it's far easier to select the line.

Yes, I am using polygons to describe areas of interest, these may be for different reasons, or at different elevations but on the same x-y area. I ran into a similar problem with selecting geopoints that were very close and managed it in that case by progressively filtering them out of the choice list as they were selected.

17

Ah interesting! Do you think you'll be using a similar technique (filtering) to tame overlapping polygons, or will that not be appropriate in that use case?

Making the inner of shapes selectable introduces a lot of weird possible scenarios stemming from overlapping shapes or points/lines running through shapes. It's probably going to be very hard to find the "correct" solution to a lot of these, as different use case will probably call for different ideal behaviours.

18

In the case where I want people to report on the condition of each item (eg a pile of previously reported findings that need a current condition report), it's not too hard to filter them out with definition updates, progressively exposing others.

If this was a larger set of items that always need to be selectable then it's possible that some might be obscured (if vertices were identical but elevation or category different then that would be a nightmare), but currently as long as they don't overlap perfectly/extremely closely, it's possible to select by area for the 'top' shape and by vertice for the 'bottom' shape.

Short of clustering that expands to show all somehow, I agree this doesn't have an obvious solution. For me, I will have to vet the elements to ensure there aren't near identical shapes that will cause selection issues if they can't be filtered.

2 Likes
19

Thank you! That's really useful context.

20

I've just sorted an old device and a testing Google account to give this a go.

I used the sample form from @Grzesiek2010 and I can report that it works as expected - being able to select the polygon from within the area is great. I can also select lines by tapping on them (not just vertices).

Changing transparency / colour of selected item would be helpful as a visual check before tapping the 'select', and probably for @ahblake situation of overlapping polygons or closely spaced lines.

Probably a whole different issue / way of implementing the select - but would it be possible, in the case of polygons / lines that share part of their geometry, for the tap to be used to list all polygons / lines that coincide with the tapped location, and then the enumerator could select the appropriate one based on attributes shown in that list? Could be a very specific use-case that causes a very general headache to solve?

Great work on this, thanks to all involved. I have a live project that would benefit greatly from the use of geotraces :slight_smile: and have just completed one that would have been easier with polygons.... Ah, the impatient users!

2 Likes