Hello,
we are using external GPS-devices for better accuracy (like EMLID REACH II) and connect those via Bluetooth to our field tablet using an app that sets the mock location. We're now facing serious precision issues since the feature to wait for the accuracy threshold is not working any more. The precision in ODK collect is always set to zero for mock locations. I read why you have made this design decision (Restricting fake location application while collecting gps points with Collect android app) and would like to ask to overthink this. It would be great if you could specify the behaviour within the form. Additionally, it would be great to offer a feature that allows to average the location for some time (5 sec.) for better precision.
Best regards,
Julian
@JulFrey Welcome!
I am looking into this option and realise that this creates a problem if using an external GPS. I also came across this with an enumerator whose inbuilt GPS stopped working so he attached a bluetooth GPS - when I asked why he collected all the data manually we discovered the problem... and I had to apologise!
As a potential (and hopefully simple) solution, would it be possible to for ODK Collect to signify mock-locations with a negative number rather than simply 0. From the post you found It sounds like there is a way of flagging the use of mock location by isFromMockProvider() so if ODK Collect can set it to zero, maybe it can be converted to negative instead? Hopefully @LN or @seadowg or @Grzesiek2010 can confirm? 
If so, it could be the 'best' of both worlds - the accuracy is important when mapping so just seeing zero creates a problem for quality assurance purposes - at present it is not possible to tell if it was manual or external / mock-location. This would provide rich dataset with no additional 'overhead', rather than assuming external GPS is false data...
Seeing a negative value means that you can ask the enumerator for evidence of an external device and then review or accept accordingly... And on import to any 3rd party software remove the negative status to make it meaningful...
This would be helpful to me.
@JulFrey @seewhy I'm not sure if external GPS devices were considered when we chatted about setting mock provider locations to zero. I'm personally surprised that external GPS devices end up resulting in Collect thinking the location is from a mock provider (although it does make some amount of sense). Would you be able to share examples of the forms you're having problems with? That'd be helpful to make sure we're all on the same page. Examples of the GPS devices that you're using and info about companion apps they might use would be helpful as well!
It would be great if you could specify the behaviour within the form.
@JulFrey There might be a way to detect that the location is from an external device and we could potentially avoid having to make changes to Collect and XLSForm.
As a potential (and hopefully simple) solution, would it be possible to for ODK Collect to signify mock-locations with a negative number rather than simply 0.
I can't see any reason that wouldn't work. @Grzesiek2010 is there no other way you can think of to differentiate between manual and mock locations?
@seadowg
Thanks for your considered response.
It's way beyond my competence, but I think that in order to get Collect to read external GPS data Android needs an app running in the background that can set mock locations - I don't see any setting within Collect that allows me to select location provider.
I linked up an old bluetooth GPS unit to my phone last night and couldn't get Collect to 'natively' get my location from bluetooth if I switched off my phone's GPS. I think I will need to download a 3rd party app that can feed data - and that will come through the mock location 'portal' as far as I can see. But like I say, beyond my competence! There might be another way to do this?
Once I have got the old GPS unit connected I'll share the results... The form being used has placement-maps geopoint, and this shows Accuracy as 0 when my colleague is using his bluetooth GPS (I don't yet know the model, but I think it is a standard unit, definitely not GNSS / RTK type) - the same form works fine with internal GPS. I'll create a simple, sample form to test.
One thing that might be worth thinking about if making any changes to how location data is read from the provider: to account for high accuracy GPS it would be good to record the time of location within the string (which needs to be satellite/ location provider timestamp, rather coming from the enumerator's device). This would (hopefully) allow corrections where getting RTK (real time kinematics) to work is more challenging (e.g. in remote terrain where there is no 2G let alone 4G signal). Again way off my spectrum of competence, but I think that it should be possible to manually do PPK (post processing kinematics) if the data is timestamped - not a problem that ODK needs to address, just facilitate. I haven't worked out how to do this, yet, but given the complexity of setting up base stations in mountainous terrain, I think I might need to! It would be a shame to miss an opportunity to cover all eventualities in one pass... I understand that this might have bigger implications for existing forms and/or other parts of the ODK ecosystem.
Putting the timestamp within the dataset might also help auditing in situations where enumerators are 'post processing' location data...
Thanks as always for being open to suggestions from the user community.
Hello together,
@seadowg am using 'GPS connector' app to set Bluetooth or USB GPS devices as mock provider. In my case this is an REACH RS2 (https://emlid.com/reachrs2/) but I also used cheaper but relative precise GPS boards for drones like a Drotec DP0801, which is better than the smartphone gps sensors by far. I attached a simple form which uses locations like I use it, but its really nothing special:
Ground_Vegetation_Survey_0_2.xml (8.2 KB)
@seewhy Regarding the differential GPS or postprocessing options: This can be done using payed third-party apps like PPM commander which also provide a mock location. Therefore I think ODK-collect is the wrong address to cover this in my opinion. If you want to use post processing you will need much more than the location and the GPS-Time. I think this is out of the scope of odk.
Having negative precission values would be fine for me, but a feature to average the location for a few seconds would be really great.
Thanks for the feedback on location accuracy and equipment - as you can see, once again a little knowledge is a dangerous thing - I get an idea and run, run, run...
I agree that ODK would not address the post-processing issue, but I thought that if the timestamp was recorded I could cross-refer to the corrected dataset when I need that level of precision. Otherwise, as you say the whole range of data sent by the gps would need to be stored in ODK - well out of scope! I am always looking for 'work arounds' where a real solution is highly complex!
Speaking of which, would it work if you were to exchange the geopoint for a geotrace field, set the interval of recording points and then you can do that average calculation for each geotrace to turn it into a point or does the use of the map appearance in XLSForm allow that 'settling' of the location to occur, close enough to averaging?
Would it be possible to start another feature discussion around averaging? I'd like to keep this thread focused on external devices if possible.
I'll have a play with this app (and look at some alternatives). Before looking into adding something like a negative accuracy, I'd like to understand if there's a way that Collect could just grab the location data from the external device. Wishful thinking: Android might handle a lot of the work of communicating with these devices for us.
What would be really helpful is if someone could recommend a cheap (<$100) device that we might be able to use to test with.
Hi @seadowg
I will send you my old iBT-GPS unit (model 737) - still available but still cost about ÂŁ100 new. Mine sat neglected in a cupboard for 5+years (used to connect it to a Pocket PC, back in the day!) and I got it out the other week to test this external GPS issue. I have just bought a Sparkfun Express RTK (similar architecture to the REACH RS2 I think) so won't need the unit in the near future. The principle should be the same (i.e. device independent) but not the accuracy.
Being able to 'natively' detect the external GPS with Android doing the heavy lifting would be good!
I have downloaded Bluetooth GPS which is on Play Store by GG MobLab - it works fine with both units and Android 6, but reviews indicate problems with Android 10 (same for other apps I think).
Thank you so much! That'll be really helpful.
Hi @seadowg
I found an alternative app called Bluetooth GNSS from ClearEvo.com (on the Play Store). It can handle NTRIP / RTK corrections (irrelevant for the old iBT-GPS), works as a mock-location provider and is open-source (no ads!). Works well with ODK Collect.
I noticed that using the placement-map option, the mapbox mapping is able to detect the accuracy of the external gps by putting the blue circle around the point (when I have 1m+ accuracy). So just need to get the accuracy data recorded in Collect... But being mindful of people having to cope with naughty enumerators making up their locations!
Thanks for your help.
I am sorry, for the late reply, I was on vacation. Handling external GPS devices is device specific and might require a proper configuration of the device. Basically, it should be possible to parse NMEA, RINEX or UBX (raw GPS device) messages directly in ODK collect, but I guess the overhead is relatively large. Especially since the way the mobile device is getting the position data could vary between Bluetooth, USB or WiFi. It would also require additional configuration fields in the ODK collect settings to select the GPS device if there are multiple devices available (internal and external). Therefore, I think using third party apps with the additional options for RTK/NTRIP and mock providers is the fruitful route. Maybe it is helpful to take a look to RTKlib (http://www.rtklib.com/) to get an idea how complicated all those things are.
There are two things that need to be solved in my opinion: (1) the accuracy of the external gps needs to be considered when placing points within the precision limit in the collect app and (2) that it is detectable that a mock location provider was used in the data (aggregate / central). I think it would be enough to simply take the precision from the mock provider for the point placement in collect and multiply it with -1 prior saving as @seewhy suggested.
I haven't had a proper chance to look into this, but I imagine you're right that Collect would have to interface with the devices directly. I had hoped there might be a generalized way to work with external GPS in the Android OS (like with microphones) but that hope is pretty weak.
Agreed. I'm going to have a play with the device that @seewhy and then come back with a potential direction.
@JulFrey @seewhy I just wanted to double-check anything to make sure I'm not making any dodgy assumptions here. To set up Collect to use an external GPS device (in the current or older versions of the app), you need to:
- Download an app that can provide a mock location (Bluetooth GPS is an example, Emlid recommend Lefebure NTRIP Client for their devices)
- Enable developer options on the phone
- Enable the downloaded app as the "Mock location app" Android's developer options
- Pair the phone with the external GPS device and set it up in the app
Does it sound like I've gone through the same steps you/your enumerators do?
Yes, this is exactly how I did this, besides the fact that I used an different app (https://play.google.com/store/apps/details?id=de.pilablu.gpsconnector&hl=en_US&gl=US) for the mock lacation.
Apologies for the lack of response - I've been out doing remote fieldwork near Assynt (no signal for a week - blissful!) - mock vacation!.
The work-flow you describe is how I did it - and from this you can see that it is very much a deliberate choice, rather than accidental to end up with mock location as a source, so should be easily verified if anyone is concerned whether their enumerators use external GPS... I suspect in a few cases it would need some support to set up?
Thanks.
I've spent a bit of time playing around with the Bluetooth GPS device (that @seewhy very kindly sent to me) as a location provider when using Collect. As discussed here, that involves using the "Mock Location App" developer option in Android - I've also verified that this feature still works on later versions of Android (including Android 12). Of course, enabling this feature will cause Collect to ignore the accuracy reported by the external device and default the accuracy for points recorded to 0. This was intended to prevent any shenanigans around faking locations when filling out forms.
Here are a few options that come to mind (some of which have already been mentioned) of how we can re-enable collecting accuracy for enumerators using external GPS devices with Collect:
- Stop setting mock location accuracies to 0 - simplest thing to do but would reintroduce the problems some projects were experiencing with enumerators faking locations.
- Annotating the location data in the filled form to flag the mock location - would allow us to preserve the accuracy for mock locations, but I believe anything we do here could cause problems for parsing submissions for analysis and would disrupt anyone already using "0" to detect fake locations.
- Make mock location accuracy negative (5.64 becomes -5.64) - seems less risky for parsing than the above option but still causes disruption for fake location detection.
- Add Collect setting to enable negative accuracy - allows us to fix this for external devices without disrupting existing projects but does mean maintaining a Collect setting (potentially long term).
- Introduce a parameter for "geo" questions that describes how to deal with mock locations - allows mock locations to be enabled by projects but would require changes to more of the ODK ecosystem (and to forms being used in the field).
- Allow external GPS devices to interact directly with Collect - would be ideal but requires building and maintaining integrations with several communication protocols as well as UI to configure devices.
After discussion with the team, it seems like option 5 is the favourite. Option 3 was also considered but there were worries around the risks presented to existing data analysis pipelines by introducing negative accuracy (and to projects already using 0 to catch mock locations).
I propose we introduce an XLSForm parameter for geo questions (geopoint, geotrace and geoshape):
| type | name | label | parameters |
|---|---|---|---|
| geopoint | location | Location | allow-mock=true |
When the allow-mock parameter is true, the client should allow “mock” locations. In Collect, this will mean that locations from Android’s mock providers will use the reported accuracy. The default for allow-mock will be false so this behaviour will be “opt-in” and by default, “mock” locations will still have an accuracy of 0.
A nice advantage of this is that this “filtering” of mock locations will be a more explicit feature of ODK as it will be the documented behaviour when allow-mock is false.
This sounds like my prefered sollution. I would only be carefull with the name. allow-mock sounds like false would disable the possibility to use mock location providers compleatly. Therefore it should be something like allow-mock-precission. Thanks a lot for taking care about this.
Hi @seadowg
Thanks for your work on this.
From my perspective option 5 would work, although I think that would mean re-deploying all existing forms so that I can allow external GPS providers to be used. Not a problem as I have a relatively small number (<20) of 'live' or 'dormant' forms.
My concern would be (if I had shenanigans going on) how to differentiate between 'real' external GPS and mock-location (I think @JulFrey was making that point in his response) with option 5 - it sounds like this 'opens' mock-location feature, but doesn't flag that a mock-location is being used?
An advantage of a negative value for accuracy (option 3) is that it allows the 'administrator' to spot questionable data (currently I am guessing that any zero-value dataset is flagged manually or by script, which should be pretty simple to adapt), and differentiates that from manual data (genuinely zero-value) or internal GPS data. Still not sure it's the 'best' solution, but it gives the nuance and means that administrators could use an 'if -ve multiply by -1' script once they have assessed the veracity of the data (could that even be introduced as a new feature of Central in the data validation process?)
Another option (um, number 7?) would be to use the audit log (is that always automatically recorded?) so that if Android detects a mock-location provider, it triggers a field/ variable within the audit log for that form, but records all mock or internal GPS locations with their accuracy and leaves manual locations as 0. That would be an 'easy' flag, and not affect the data collected. Administrators concerned about false data could look for that flag in the audit - which should be as simple as the current 'look for zero', but still gives the nuance of differentiating manual locations (assuming maps-placement appearance is in play). The audit log would not work for manual transfer of data (i.e. via Briefcase on a PC with a cable!) but then you'd be likely to have an idea whether that device had external GPS capabilities set up...
Both of these options have zero / low impact on already deployed forms, I think, making either backwards compatible. They do require notification to anyone relying on 0 to catch the naughty enumerators, so are not without 'cost', but it looks like option 5 isn't 'free' either...
However, we need to check it works for those who are concerned about abuse of mock-location providers, so should probably involve them in the decision-making process (to avoid the pendulum swinging away from their comfort zone). Quick shout out to @Lama , @caneeraj, @arcunha and the redoubtable @Xiphware in case they have any further thoughts. Seems like the issues were raised in one way or another, now I look back at the 'cause' of the zero-accuracy setting. Not wishing to be critical of that discussion, it appears to have looked at solving their immediate problem without picking up on the current consequences - so I am mindful of finding something that works for everyone (including you lovely people at ODK!).
Well that's thruppence of my tuppence (or 3 of my 2 cents) worth on this topic! Hope it helps...
Good point. How about allow-mock-accuracy to make it clearer?
As far as I understand, we haven't really heard a need for both detecting mock locations and using external devices/mock locations legitimately. The assumption I have in my head is that programmes/projects that need to use external devices are not concerned about locations being faked/mocked - this could, of course, be incorrect.
Another option (um, number 7?) would be to use the audit log (is that always automatically recorded?) so that if Android detects a mock-location provider
I personally like the idea of this being added down the line! It feels like a simpler way to deal with detecting mocks for projects that might also want to enable their accuracy (the case that would destroy my assumption above) than needing to flag the mock value using the value itself.
If you put the effort into this question to solve this problem using an option in the form, I would try to solve it compleatly for the future. I agree that the default should stay the same. I would add "mock-accurancy" as a field and the add the options "false" (default), "true" and "force". The last option would set the precission to a defined unrealistic value to avoid the accidential use of internal gps sensors in cases when a higher precission is needed. I would prefer something like 9999 as a precission value to make sure the user recognizes that something is wrong. True would just handle internal and mock-gps providers the same way.