Possibility of an endpoint to trigger pending uploads to S3

Development
1

Problem

  • There is a cron that runs every 24hrs to upload files to S3 (if configured).
  • I have overridden this in a Dockerfile to run every 15 minutes RUN sed -i 's/^0 5 \* \* \*/\*\/15 * * * */' /etc/cron.d/odk
  • However, it would be great if this can be triggered via an endpoint in Central instead:
    • Opens the possibility to trigger the uploads via a webhook, or other action in an upstream application.
    • I have a library that is using pyodk and as part of the tests I upload some submission attachments and list the S3 URLs. However the URLs aren't present yet as the files aren't uploaded. Running with docker, there is no way to trigger the uploads via subprocess or similar. The only way I can see is via API call from another container.

Possible solution

  • Add an endpoint /v1/projects/{projectId}/forms/{xmlFormId}/submissions/{instanceId}/attachments/s3-sync or similar. Update: this should probably be a system endpoint /v1/sync-s3, as it's not specific to a project / is system wide.
  • The endpoint simply runs lib.task.s3.uploadPending, uploading pending files to S3.
  • Probably no need to return a response body? Just an immediate HTTP 200 that the upload request was triggered.

Is this useful?

  • I wanted to gauge is this is useful and/or something the ODK team is willing to support.
  • Seems like a small bit of code with minimal implications.
  • Happy to contribute this if given a green light :+1:
1 Like
2

Small update to this - I added a proof of concept here (migration + router):

Adding it to the Central container like so:

# Add custom code s3-sync endpoint
# https://forum.getodk.org/t/possibility-of-an-endpoint-to-trigger-pending-uploads-to-s3/54110
COPY s3-sync/20250313-01-s3-sync-permission.js /usr/odk/lib/model/migrations/
COPY s3-sync/s3.js /usr/odk/lib/resources/
# Modify the lib/http/service.js file to include our new router
RUN sed -i "/require('\.\.\/resources\/oidc')(service, endpoint);/a \ \ require('../resources/s3')(service, endpoint);" lib/http/service.js
1 Like
3

We have been using this code in production for a few months and it works well.

We found a bug upon doing some load testing, in preparation for 500 users submitting in parallel - fixed now.

I made a PR based on the code: https://github.com/getodk/central-backend/pull/1494

If it's useful to anyone please thumbs up! :folded_hands: