QuODK - another connector between ODK Central and QGIS to get your data more easily!

I have just released a new plug-in for QGIS that allows you to download data directly from ODK Central using the API… The idea is to cut out the middle-person so that people can download data without needing to log into Central, or be familiar with API scripting. This allows a QGIS user to analyse ODK data without the intervention of a Data Manager or use of a dashboard…

https://plugins.qgis.org/plugins/quodk/

So that's Qgis > yoU < ODK central (QuODK or maybe Quo DK!)

Credit where it is due, I was inspired by FooODK and have borrowed some of the basic functionality, but got carried away with adding functions for my needs!

You need access credentials for your ODK Central server before starting. These are saved (in plain text format) in the plugin’s folder. You can choose whether or not to save the password.

IMPORTANT
ODK Central Administrators might want to think about what access to provide – Data Viewer works fine, and bear in mind that this user will have access to any Projects given to that user in ODK Central. So that includes all forms within each of those projects, and all submissions within each of those forms. If you think this will / could cause a problem, please do not give ODK Central credentials to people you do not trust (or think might just be too nosey for their own good) - currently there is no way to change the data on ODK Central using this plugin, but if the data is sensitive it could be a privacy issue that you need to consider. (NOTE TO ODK CORE TEAM – this is why I would like more granular control within ODK Central for web users!)

USE
The plug-in looks a bit daunting first-time round, but hopefully if you can follow the logic, it should become fast and simple… Honestly! There is a cascade of drop-down selects to help you narrow down the submissions (and/or repeats) that would want to access. And any that have just one value become defaults to speed things up (so if you only have access to one project, it will populate the list of available forms automatically).

Here's a screenshot of the plug-in:

Once connected to the Server, select one of the project’s you have permission to access, this triggers the list of forms. On selecting a form, QGIS will connect to ODK Collect via the API and retrieve a set of submissions within the date range (and store them in memory as a dataframe). You can then filter on any attribute to narrow the range. The table is shown to help you check before loading data to the canvas.

This is an example of populated form:

The geometry column is derived from any of those in the form (shown as question name + geometry type) and you can transform the data to your preferred Projection before loading - it also appends a field called geom_from_ODK. If there is no geometry column, you can choose to load the table only. I chose to use the 'native' ODK format for geometry and convert that rather than call for WKT - this is important for my work where accuracy of data collection can be useful.

You need to load a separate layer for repeats (again with or without geometry). The relationships will be KEY / PARENT_KEY if you need to associate them (the same as CSVs downloaded directly).

These will be loaded as temporary layers in QGIS and you can export (geopackage is good!) or copy and paste the features into existing layers that have some or all of the same attributes (only those attributes in the ‘host’ layer will be pasted, which can be an advantage or a disadvantage!). Other QGIS plugins can be helpful to avoid duplicate values when appending features to existing layers, if you end up with overlapping dates (try Append Features To Layer, for example!).

If you prefer to download the data as CSV you can select that option, and depending on the geometry you can load it to the canvas (delimited text layer) – but only with lat/long (EPSG:4326) at the moment.

You can also download the attachments to a local drive. QuODK will create a new project variable called @ODK_image_path so that you can view attachments in attribute forms more easily (HINT: in QGIS use that variable as the ‘default path’ with a widget type Attachment and the relevant media type) – bear in mind that it will overwrite this Project Variable value each time you download via QuODK, so this might not be very useful if you store images in different folders for use in the same QGIS Project…

I confess that this has significantly reduced my Admin of ODK Central by allowing other team members to get their own data, and for us to grow datasets easily during data collection phases. However, it is only appropriate in teams where the data is not confidential or without restricted access - that’s beyond my pay grade and the API doesn’t currently allow those filters “upstream” as far as I know - happy to update it if there is a way to do that securely.

Plenty of scope for improvement and I’m thinking of adding Entities to the functions, but this is not imminent…Apologies for the kindergarten level python coding! Translations should be possible, but it's not a fully 'compliant' method - there are a core set of phrases in the head of the code - sort of a relic of adapting FooODK (my Swahili is not very deep!). NOTE: this does not work with any other flavour of server (e.g. Kobotoolbox) and I currently have no intention of adapting it as I would like to focus on contributing to ODK directly...

I thought this would be more useful to the ODK community than being available only to my clients…I hope it unlocks some more use cases for ODK too! The usual caveats of 'buyer beware' apply - don't hold me responsible for anything that goes wrong :slight_smile:

7 Likes

Thanks a lot Chris ! Will try it as soon as possible.

A quick Twixmas update to the plugin...

I've now changed the filter so that you can either include or exclude submissions based on a variable. Seemed to be a logical thing that I'd forgotten :slight_smile: So you can now exclude submissions that have the status Rejected, for example.

I have also dropped the Experimental Tag in QGIS - which might be a rookie mistake.

Happy 2025!

3 Likes

Looks absolutely amazing, great work!

Probs worth a quick check of the branding guidelines re use of ODK - QuCentral might be an acceptable name, use of ODK and logo might not.

I'm saying this with burning ears, having hogged ruODK (and gotten special permission from TAB back in the day).

1 Like

Ah, good point, well made.

Happy to change the name and remove my doodles of rebranding - absolutely no offence intended - it is homage rather than appropriation!

I nearly have a version that can download entities from Central directly into QGIS so hopefully that might be a logical opportunity to rename and debrand? I was hoping to be able to finish developing the creation of geo-entities before updating, but that can wait... As it requires different permissions on Central, maybe that is a good excuse to release another version without that functionality.

I hope it is a reasonable question on how people might easily find the plugin if it is called QuCentral or some other conflagration of QGIS and Central, without ODK appearing. I'm not being defensive, more curious - I am mindful of branding but also see the constraints of using (or not) names.

Hopefully the branding lawyers will pick this up and send me the relevant cease and desist notification :slight_smile: Hopefully without any public shaming (or too much)... Private messages gratefully received.

I agree the keyword ODK helps a lot to find/spread, test and maybe use a tool.
Maybe something to think about for community contributions : how to increase the visibility of community contributions without unwanted side-effects on ODK brand.

@seewhy I still have to test your extension !

1 Like

In case anyone else makes a rookie error - the server url is everything before /#/projects, not ending in /! (Took me longer than I like to admit to realise that)

For attachment download - it doesn't appear you can cancel partway through? While it's in progress I can't return to the plugin dialogue and QGIS itself has a spinny wheel.

edit: I went away for a couple of hours and came back to find it still going - I had a date filter but it seems to be attempting to download all attachments - is that expected behaviour?

Keen to see the entity compatible update, great work @seewhy!

Thanks for the feedback @ahblake

I've added a check on the '/' to automatically remove that from the URL, to save further blushes - that will be in the next release...

For attachment downloads - as with processing dialogs etc with QGIS you can't do anything else while the plugin is actually running (doing something, not just open). It took me a while to work out how to add a functioning progress bar instead of the 'not responding' spinner of doom - but that also freezes if you try to return to the canvas (as do most others').

The plugin SHOULD only download the attachments related to the selected submissions (there is a message bar at the bottom of the dialog that tells you how many submissions and attachments it has found). So it's not expected behaviour for it to download every attachment on the server unless you ask it to - that was one of my must-have features of this plugin, to save repeatedly downloading attachments. If you can screenshot or otherwise help me understand what happens, I'll look into it. I use the plugin a lot and haven't managed to do that. Believe me, I'm usually the one that breaks things!

The table should also show you the submissions (or repeats) that it will be loading to canvas / CSV - was there a discrepancy there too with what you were expecting? And did you find every attachment in your selected local folder?

Did you, by chance, tick 'ignore dates'? That overrides the date filter and therefore includes all submissions for the selected form (maybe I need a trigger to automatically switch that off again if the dates are changed?)

I can't recall if i ticked ignore dates, I'll try again tomorrow (on a much smaller project!) , it's finished downloading every single attachment but i can't switchback to the dialogue or qgis proper, looks like I'll have to force quit.

Oh no! That was exactly what I was trying to avoid with this plugin. QGIS usually starts to respond once the downloads have finished and then you can close the window... Maybe it doesn't like reading code upside down :slight_smile:

Let me know what happens and I'll look into it.

Was able to show all windows and see the dialogue and you're correct, I had accidentally checked ignore dates - it was the behaviour below that probably fooled me. It and QGIS were still spinny wheel of death 24hrs later so needed to force quit.

I'll try again and see if I can get a successful partial image download without going non responsive.


Does the filter use 'submitted at' for dates in UTC or other TZ? If I filter Central 'Submitted at' I can get different counts to the same date window in the plugin which appears to be related to my local offset. Checking one record I have a __system.submissionDate as 2024-12-11T05:57:54.165Z and a Central displayed date as 2024/12/11 16:57, so it does appear that the filter is UTC vs my UTC+11.

For a smaller filtered set of submissions;

  • Load layer to canvas works.
  • If I try to save CSV I get an error in QGIS and no output file, progress bar continues to pulse but I can change filters and reset it. Error was OSError: [Errno 30] Read-only file system, this was the same folder that the attachments successfully downloaded to. Changing folder location worked, changing back then started to work... :person_shrugging:
  • If I click save as CSV and then cancel, I get a 'No data - try changing date range' error, but there's records still showing in the table
  • Download attachments fetches all attachments, not just those in the main form / selected repeat, successfully completed and showed "# downloaded to \path" and 100% in the progress bar.

I can force some unexpected behaviour with the ignore date checkbox;

With my date filter still in place, selecting 'ignore dates' didn't change the content of the table or the 'Number of submissions found' value, changing the 'select sub/repeat group' did then update to show every submission. Then deselecting 'ignore dates' didn't reapply the filter, and loading layer to canvas brought in all 10K submissions.

With a tight date filter but all 10K present using ignore dates, then deselecting it and then applying a value filter (exclude rejected) it still used the entire dataset, I had to change (or reselect) the subset value (main/repeat) to get the date filter to apply again, although with a funny message as 'Number of submissions found: 21 (50930 attachments)' that has a value from filtered and unfiltered. Changing the date range will also reapply the date filter correctly after deselecting ignore.