Automatically synchronise media files from ODK Central to a webserver

Development
1

Hi

What would people suggest is the best and simplest way to synchronise media files from ODK Central to a Webserver? This would replace the current manual way of doing things which are to download the zip file from ODK Central, Unzip it and then upload it to the webserver via FTP.

Synchronisations could be on an hourly (or similar) schedule or interval

Thanks

2

Pull data on a remote system via Central API should do the same, no?

1 Like
3

Thanks Saad

What would you say would be the easiest way to do this, would it be via a cron job on the server or on a third part connector service such as make.com?

Thanks

4

Hi,

I haven't tested any online service for this yet. Mostly custom dashboards I make can pull data from ODK Central via API, and store locally (AWS EBS disk, S3 storage, etc). Of course, you can set any frequency for it as per your wish.

However, please do note that this is mostly far-end triggered (meaning the dashboard will make a request to Central to pull data). Central itself would not be doing it.

Regards,
Saad

5

Hi @ODK-MAC,

To synchronise data (together with media files) to a server, I've built scheduled ETL pipelines using R, Docker, cron. A high level overview is here with the two salient bits being

image
image1920×1200 150 KB

and
image
image1920×1200 346 KB

I found it far faster and more maintainable to write out the data mapping step by hand (broken into many R functions, one for each table in the target database) as opposed to using a third party tool.

If Python is more your jam than R, you could build the same architecture using pyodk.

If you only want media files tro be downloaded to a certain directory, your script could be as simple as ruODK::odata_submission_get(local_dir="path/to/your/dir") which downloads and parses submissions together with all media files. It skips existring downloaded media files, so any repeated run only fetches new media files.

1 Like