Geowidgets - different representation / invalid(?) WKT / capture-accuracy parameter error

[ahblake]

1. What is the issue? Please be detailed.
ODK Central 1.5.2 via ODK Cloud
ODK Collect V2022.2.3

Item 1
I was looking at the geowidget values and noticed they are different depending on where they are stored/queried from,

• lat/long (raw/CSV) vs long/lat (Central/OData)
• discarding accuracy except in CSV
• adding delimited labels to points in CSV
• different number of significant figures for manual vs GPS
• SRID prepended to string and manual point elevation converted from 0.0 to 0 in OData

I found the Central WKT values are invalid when tested here but not here. One requires a Z if elevation is included, one won't accept a Z and neither accept a string prepended with SRID=4326;

Points:
Raw value in form: latitude longitude elevation accuracy using a note to display the string on screen, under the View/Change button it's in DMS and 1/2 dec places

• Manual placement
  • -25.58161926676068 134.81799200177193 0.0 0.0
• GPS acquired (fewer sig figs)
  • -25.5816192 134.8179920 90.30000305175781 13.451000213623047

Displayed in Central: POINT (longitude latitude elevation) Not a valid WKT string, should be POINT Z? precision not shown but indicated by # of significant figures?

• Manual placement
  • POINT (134.81799200177193 -25.58161926676068 0)
• GPS acquired (fewer sig figs)
  • POINT (134.8179920 -25.5816192 90.30000305175781)

CSV download: delimited into separate fields with appended label; ${geopoint}-Latitude,${geopoint}-Longitude,${geopoint}-Altitude,${geopoint}-Accuracy

• Manual placement

  • ${geopoint}-Latitude,${geopoint}-Longitude,${geopoint}-Altitude,${geopoint}-Accuracy -25.58161926676068,134.81799200177193,0.0

• GPS acquired

  • ${geopoint}-Latitude,${geopoint}-Longitude,${geopoint}-Altitude,${geopoint}-Accuracy -22.8328141,127.2509277,90.30000305175781,19.222999572753906

OData query: SRID=4326;POINT (longitude latitude elevation) Errors in both WKT tests, what is this format called when prepended with SRID? Manual elevation converted from 0.0 to 0

• Manual placement
  • SRID=4326;POINT (127.1663636341691 -28.224008368775536 0)
• GPS acquired
  • SRID=4326;POINT (127.161421 -28.2213811 61.400001525878906)

Traces

Raw value in form:

• Manual placement: latitude0 longitude0 elevation0 accuracy0;latitude1 longitude1 elevation1 accuracy1
  • -12.111167619268736 135.51907416433096 0.0 0.0;-34.953320843366264 136.2364909797907 0.0 0.0
• GPS acquired (fewer decimal places on lat/long, accuracy has 3 dp)
  • -12.1111676 135.5190741 90.30000305175781 6.444;-34.9533208 136.2364909 90.30000305175781 6.444

Displayed in Central: LINESTRING (longitude0, latitude0, elevation0,longitude1, latitude1, elevation1) Not a valid WKT string, should be LINESTRING Z? precision not shown but indicated by # of significant figures?

• Manual placement
  • LINESTRING (135.51907416433096 -12.111167619268736 0,136.2364909797907 -34.953320843366264 0)
• GPS acquired (fewer sig figs)
  • LINESTRING (135.519074 -12.1111676 90.30000305175781,136.236490 -34.9533208 90.30000305175781)

CSV download: latitude0, longitude0, elevation0 accuracy0; latitude1, longitude1 elevation1, accuracy1 space added after ;

• Manual placement
  • -28.22388624497617 127.17291642264377 0.0 0.0; -28.223896146965686 127.172913288845 0.0 0.0

OData query: SRID=4326;LINESTRING (longitude0 latitude0 elevation0, longitude1 latitude1 elevation1) SRID prepended & elevation changed from 0.0 to 0

• Manual placement
  • SRID=4326;LINESTRING (127.17291642264377 -28.223886244976171 0, 127.172913288845 -28.223896146965686 0)

Shapes

Raw value in form: latitude0 longitude0 elevation0 accuracy0;latitude1 longitude1 elevation1 accuracy1 latitude2 longitude2 elevation2 accuracy2 latitude0 longitude0 elevation0

• Manual placement
  • -11.673939274410756 142.7346308156848 0.0 0.0;-34.38366691984674 116.43875923007727 0.0 0.0;-37.63004505371219 149.01771750301123 0.0 0.0;-11.673939274410756 142.7346308156848 0.0 0.0
• GPS acquired (fewer sig figs on lat/long, 3 on accuracy)
  • -11.6739392 142.7346308 90.30000305175781 15.977;-34.3836669 116.4387592 90.30000305175781 15.977;-37.6300450 149.0177175 90.30000305175781 15.977;-11.6739392 142.7346308 90.30000305175781 15.977

Displayed in Central: POLYGON((longitude0, latitude0, elevation0,longitude1, latitude1, elevation1,longitude2, latitude2, elevation2,longitude0, latitude0, elevation0)) Not a valid WKT string, should be POLYGON Z? precision not shown but indicated by # of significant figures?

• Manual placement
  • POLYGON ((132.7346308156848 -21.673939274410754 0,116.43875923007727 -34.38366691984673 0,149.01771750301123 -37.63004505371219 0,132.7346308156848 -21.673939274410754 0))
• GPS acquired
  • POLYGON ((132.7346308 -21.6739392 90.30000305175781,116.4387592 -34.3836669 90.30000305175781,149.0177175 -37.6300450 90.30000305175781,132.7346308156848 -21.673939274410754 0))

CSV download: See geotrace

OData query: See geotrace

---

Item 2

I noticed a field user was not manually entering a point as requested as there were altitude and accuracy values in their submissions, I tried to use parameters to force manual selection per the docs but this doesn't work as I expected in a few ways:

1. For a geopoint with appearance as placement-map, if the GPS is fast / already has a good fix, it may record a location before the enumerator can select one. I tried to prevent the enumerator from having to clear it and then select the point, by setting the accuracy to an unobtainable value under parameters; in this case capture-accuracy=1 warning-accuracy=1 allow-mock-accuracy=true, but this had no effect and a value was populated with accuracy of ~15m within seconds. This then submitted with no warnings.
2. For a geotrace or geoshape the capture-accuracy and warning-accuracy parameters are not allowed, with this error showing if they are used on either;
   
   
3. For a geopoint, geotrace or geoshape the allow-mock-accuracy=false doesn't work as I expected, when a location is manually chosen, so the accuracy = 0.0, this submitted with no warnings.

2. What steps can we take to reproduce this issue?

Item 1

1. Compare raw field value in form (display with note) to the same value in Central, CSV download & OData query
2. Test WKT outputs in WKT Playground and WKT Sandbox for validity

Item 2

3. Test a form with accuracy parameters to see if they are accepted on form validation (trace/shape) or if they act on the collected values (all)

3. What have you tried to fix the issue?

Item 1
For the difference in CSV/Central/OData I currently have to ensure that I only use one source to get values and transform them into the desired / correct format

Item 2
For preventing point autopopulation I have no solution currently.

For forcing a manual location input, currently to try work around this, I have set constraints instead of using parameters, so if you want to force the user to manually select a geopoint with appearance as placement-map, you can use a constraint to ensure the accuracy value is 0, for a geopoint this is
selected-at(${geopoint}, 3) = 0

However for a geotrace/geoshape, as the resulting string is space delimited but each point is ; delimited, you need to look before the ; so the constraint is (note; this only checks the first point in the line/shape):
substring-before(selected-at(${geotrace_shape}, 3), ';') = 0

This is the opposite of using a constraint to require better GPS accuracy as in the docs, instead of looking for a value under a set radius, you look for it to be 0.

[LN]

I think most of the differences you've identified are differences between the various specs we try to follow.

@ahblake

lat/long (raw/CSV) vs long/lat (Central/OData)

The raw data uses the ODK notation. It uses lat/long based on ISO 6709 and common lay usage.

The OData standard states that locations should be represented as GeoJSON. The GeoJSON spec specifies lon, lat order.

@ahblake

discarding accuracy except in CSV

It's not part of the GeoJSON or WKT standards. It is added as a property to the GeoJSON in the OData feed but PowerQuery wouldn't know what to do with it so I guess it doesn't display it. I can dig deeper into whether it can be retrieved by doing something like changing type and splitting if this is critical.

@ahblake

adding delimited labels to points in CSV

I'm not sure what this references, can you please show an example?

@ahblake

different number of significant figures for manual vs GPS

When points are placed manually, the mapping engine (Mapbox, OSM, GMaps) will divide up the screen and I guess give as many digits as they can represent. For device-captured points, the coordinates come from the Google Fused location provider. I believe that the number of decimal digits you get will depend on several factors including the version of Google Play Service and the sensors available (GPS, wifi). Collect passes on these raw values which I think is the best thing for it to do. This GIS StackExchange post has really good information about the meaning of these decimals and how they relate to accuracy.

@ahblake

precision not shown but indicated by # of significant figures?

We can't know anything about precision without following some kind of calibration protocol. Accuracy is defined in the docs.

@ahblake

SRID prepended to string

Unfortunately, the SRID=4326 notation comes from the way PowerBI or Excel reads GeoJSON from the OData feed. See this PowerBI forum thread. I've done some mapping in PowerBI by splitting coordinates as suggested in that thread. It also loses the .0.

@ahblake

Central WKT values are invalid when tested here but not here

Is there a tool that you'd like to use WKT values with but can't because of this? Will look into it.

@ahblake

I currently have to ensure that I only use one source to get values and transform them into the desired / correct format

I'm afraid I don't see a great alternative! My experience is that folks build their pipelines around one source. It is particularly annoying that some don't include accuracy. If there's a specific tool you're unable to use because of this we might be able to come up with a solution.

@ahblake

For a geopoint with appearance as placement-map, if the GPS is fast / already has a good fix, it may record a location before the enumerator can select one.

Here are the problems I've seen placement-map used to solve:

• The environment means it's often not possible to get a high-accuracy point but the user can use points of interest to get closer to their actual location.
• The user is not exactly in the location they want to capture. E.g. they're standing in the street and want to capture the location of the door to a specific house. The zoom to location gets them close and then they can adjust.

It sounds like you're doing something like the second case and you never want the user's current location? This relates to the request for a manual-location-only question type here.

@ahblake

by setting the accuracy to an unobtainable value under parameters; in this case capture-accuracy=1 warning-accuracy=1

Indeed, those are only supported for geopoint with no appearance. I think we should fix that and I like it as a way to address the issue above.

@ahblake

For a geotrace or geoshape the capture-accuracy and warning-accuracy parameters are not allowed

That's right. Those both have more complex configuration that's not exposed by the form definition currently. We'd have to expose at least collection interval in addition to accuracy, I believe. More at Geo: Setting an accuracy threshold and collection interval in the form definition.

@ahblake

allow-mock-accuracy=false doesn't work as I expected, when a location is manually chosen, so the accuracy = 0.0

That sounds like a documentation issue! allow-mock-accuracy=false means the accuracy value for a point that comes from a mock provider (external app such as a spoofing app) will automatically get an accuracy of 0. That's the default. There's no external app involved in your workflow so it has no effect. The goal is to be able to detect in analysis that a user tried to use a location spoofing app.

@ahblake

for a geotrace/geoshape

I think the best answer here would be for the form to allow requesting manual collection and hide the auto collection option.

[ahblake]

Thanks @LN, this helps my understanding greatly, appreciate the references/links, I'll give them a read.

You are right, much of my post wasn't 'this is broken' it was more, 'I can see differences, but I'm not sure why they are different'. I didn't want to try to setup any tool to use the values until I knew what to expect, and you're right, it would be a single pipeline, and I don't have any specific tool issues at this point.

@LN

I'm not sure what this references, can you please show an example?

Sorry, I don't think I explained this well. raw data/CSV is supplied with delimited lat,long,elevation,accuracy fields with each value labelled as "yourgeovariablename"-latitude, etc, and OData is supplied as the GeoJSON spec with all values in a single string with a label "yourgeovariablename". (again, not an actual issue I'm facing, it was a difference I observed)

So for a geopoint called ${my_geopoint} it would return as;
CSV
Column labels: my_geopoint-Latitude,my_geopoint-Longitude,my_geopoint-Altitude,my_geopoint-Accuracy value1: -22.8328141,127.2509277,90.30000305175781,19.222999572753906

OData
column label: my_geopoint
value1: SRID=4326;POINT (127.2509277 -22.8328141 90.30000305175781)

@LN

The user is not exactly in the location they want to capture. E.g. they're standing in the street and want to capture the location of the door to a specific house. The zoom to location gets them close and then they can adjust.

Yep! In many cases the area of concern to record is not where they're standing, but visible to them (or known to them but they're actually far away from it). All I was attempting to do was to prevent the point from being created and having to be deleted first. But as there's no confirm delete dialogue so it's only a single tap to remove, or they can ignore the delete entirely and long press the desired location directly. This is fine as is and I'll better educate the user! My apologies, I missed that only trace/shape has the delete, confirm clear, add, add by tapping process, and they don't ever create a point automatically.

I'll keep my constraints for forcing manual placement for all geotypes, as they work acceptably well. The user will get sick of constraint messages and quickly learn to record the location as directed for geopoints!

@LN

I think the best answer here would be for the form to allow requesting manual collection and hide the auto collection option.

The ability to hide manual location recording/automatic location recording would be an additional help, but certainly not high priority for me.