Central2pg : PostgreSQL set of functions to get data from Central

[mrrodge]

@mathieubossaert

But if your problem is "only" to get pictures from central, I'm sure a small python wrapper using pyODK could serve you the photos directly from Central instead of building all my database centric workflow.
But I will be glad to help if

Yup - that, at the moment, is my only problem! Any pointers for where to start with a wrapper like you suggest? I've no experience with python either.

Many thanks for your efforts - it's greatly appreciated.

Looking at the get_file_from_central function, it suggests that files are saved to the filesystem but I'm guessing I would need to host them somehow to get them into my reports.

[mathieubossaert]

@mrrodge

Looking at the get_file_from_central function, it suggests that files are saved to the filesystem but I'm guessing I would need to host them somehow to get them into my reports.

Yes, in our case, the folder is used and visible by both Apache and PostgreSQL (user postgres) but it could be anywhere else your reporting tool can access

[mrrodge]

OK thanks - getting the idea now and am almost there. How do you use the get_file_from_central command? It looks like that will only download one image, where I want to download all the images for the form submissions I got with odk_central_to_pg.

Also what does it do with the file names? What's prise_image?

Thanks again!

[mathieubossaert]

Let's continue the discussion with direct messages or over Github

[mathieubossaert]

We achieved to get it work. Function were not ready to accept spaces chars in the form_id.
https://forum.cen-occitanie.org/t/suivi-bourreau-des-arbres/963/6?u=mathieu
I will also improve the example given on github's README page.
Thanks @mrrodge, maybe we can clean the showcase and drop the messages above ?

[mathieubossaert]

People who used central2pg may take a look at pl-pyodk.
It allows the use of filters to pull data from central.

Pl-pyodk : use of pyODK with PL/Python PostgreSQL functions to pull data from Central into your own database

Two years ago, I spent time during the third COVID lock-down working on PostgreSQL functions to automatically pull data from Central into a PostgreSQL database. Central2PG is the result of that research. And we've been able to maintain the workflows we built on Aggregate since 2015 on Central Every time we publish a new form, we simply set up a scheduled task that uses Central2PG to automatically pull the data into a PostgreSQL database. Central2PG works very well and we never …

[mathieubossaert]

With Central 2023.4 we can now filter data from repeat table using a top level (Submissions) filter as the submission date.
central2pg can make use of it right now and allows you to get only data that were submitted since a given timestamp or date
For example, data from our main form are pulled every hour to get fresh data, without filtering, we needed to pull all the data created since the form was published (90000) !

Now we can filter subtables on submission date and pull only the 37 observations created since last known submission date

As a consequence we can increase a lot the frequency of pulling data from Central and be very close to a real time field sync between our databases or desktop tools and the field.

Thanks a lot to the ODK Team !

[steeve_kevin]

Hi @mathieubossaert i wan to get data from ODK central to my own PostgreSQL database, when I run the function odk_central.odk_central_to_pg() I get the following error:

NOTICE: la table « central_json_from_central » n'existe pas, poursuite du traitement
NOTICE: la table « central_token » n'existe pas, poursuite du traitement

ERROR: l'argument de la requête d'EXECUTE est NULL
CONTEXT: fonction PL/pgSQL odk_central.get_form_tables_list_from_central(text,text,text,integer,text), ligne 11 à EXECUTE
instruction SQL « SELECT odk_central.get_submission_from_central(
user_name,
pass_word,
central_FQDN,
project,
form,
tablename,
'odk_central',
lower(trim(regexp_replace(left(concat('form_',form,'',split_part(tablename,'.',cardinality(regexp_split_to_array(tablename,'.')))),58), '[^a-zA-Z\d]', '', 'g'),''))
)
FROM odk_central.get_form_tables_list_from_central('@gmail.com','S','*****',3,'support'); »
fonction PL/pgSQL odk_central.odk_central_to_pg(text,text,text,integer,text,text,text), ligne 3 à EXECUTE

ERREUR: l'argument de la requête d'EXECUTE est NULL
SQL state: 22004

here is how the function is configured
SELECT odk_central.odk_central_to_pg(
'*****@gmail.com', -- user
'*******', -- password
'f.com', -- central FQDN
3, -- the project id,
'support', -- form ID
'odk_central', -- schema where to creta tables and store data
'' -- columns to ignore in json transformation to database attributes (geojson fields of GeoWidgets)
);

Quel peut être le problème?
merci

[mathieubossaert]

Hi @steeve_kevin ,

let's talk in private or over Github if you're ok to investigate and fix your problem.
My gess is the curl function returns an error so PG does not have anything to transform.

And we'll be back here to explain the problem and help other users.

[Ri_Ri]

Hello,and thanks to your hard work for odkcentral2pg.

I've installed and it work great.

Is there way to launch the script with a "signal" from odkcentral, like a submission ? The cron job are great but use memory for nothing if not used.

[mathieubossaert]

Thanks a lot for ypur feedback !
And welcome to the forum. Please don't hesitate to introduce yourself on the dedicated thread
I am close to solve @steeve_kevin's issue due to curl subtilities in a windows environement...

@Ri_Ri

The cron job are great but use memory for nothing if not used.

I don't know exactly how much memory the cron daemon uses but I think it is really light on our server. That might depend on everyone's context.
Since we can filter subtables, it is lighter than ever. If one su mission occurs thé thé last time, only one submission is downloaded.

@Ri_Ri

Is there way to launch the script with a "signal" from odkcentral

Probably... PostgreSQL can notify events and listen to notifications but I am not sure it could help to triggering central2pg.

Maybe pl-pyodk in a web context and a tool like https://github.com/crunchydata/pg_eventserv would be better.

But it would need some modification on Central.

[Ri_Ri]

thank you for your reply.

I will fill in the thread soon.

My need is to display each submission as soon as it is created. I can't do a cronjob for every project every second.

My solution was to make a trigger on the submission table. I then use the odk_central_to_pg function with the form_id that fired the trigger. Regards

[mathieubossaert]

@steeve_kevin 's problem is solved and was due to curl usage on windows and the need to give the POST parameter as a file -d @data.json

[steeve_kevin]

thank you for everyone @mathieubossaert especially for your availability.

[Ri_Ri]

Hello Mathieu,

it's ok with the trigger to push data when a user submit.

The next step is to push image in a specific folder, in a trigger too, and i have a several problem with the transaction time, i can select all informations for get_attachment_from_central, but when the fonction lunch, it's too early for the database and the fonction crash...
Anyway, I have 1 question :

-You haven't the problem that it's the postgres user who save the data with plpyodk ( for the rights of the image ) ?

You know how i may save the image as another user with plpyodk?

bye

[gcorbel]

Good morning,

Thank you Mathieu Bossaert for this amazing job, I m testing central2pg with your example form ODKwaypoints
I ran your central2pg.sql in a query tool and then I ran :
"SELECT odk_central.odk_central_to_pg(
'email@domain.fr', -- user
'mypassword', -- password
'odk.gedeop.inrae.fr', -- central FQDN
2, -- the project id,
'ODKwaypoint', -- form ID
'odk_central', -- schema where to creta tables and store data
'point_auto_5,point_auto_10,point_auto_15,point,ligne,polygone' -- columns to ignore in json transformation to database attributes (geojson fields of GeoWidgets)
);
but I get the same error as @steeve_kevin

I can see that you solved the problem, can you give me more details please because I don't understand how to " give the POST parameter as a file -d @data.json"

Thank you and have a good day,

Gaëlle

[mathieubossaert]

Hi @gcorbel ,

Thanks a lot

I'm sorry I didn't take the time to document it...
I'll send you the modified script before I publish it on gitbub (at the begining of August).