External storage for blobs in ODK Central

Currently, attachments to submissions and forms are stored as blobs in a database table. It will be great if possible to store these in a network file system, or best in a managed storage solution like Google Cloud Storage, Amazon S3, etc.

1. What is the general goal of the feature?
Reduce the database size, which will greatly speed up the backup and restore operations.

2. What are some example use cases for this feature?
At the moment, my self-hosted central database is around 70GB. The nightly export I have scheduled takes couple of hours. It will probably take similar amount of time to restore in an event of disaster. I only have experience with Google Storage buckets, but if the blobs were stored there, they would be backed up and automatically versioned by the cloud provider. It will be also possible to access the files via different means, like authenticated urls, or short-lived public urls.

3. What can you contribute to making this feature a reality?
Coding (if enough guidance given) and testing for the Google Cloud Storage implementation.

Hi @punkch! This feature is on the roadmap, and we believe that moving blobs out of the database would have several benefits. However, we also think that implementing this feature would be a large effort, requiring changes not just to the API server, but also to the frontend and our Docker setup. We're working hard on the upcoming v1.4 release and don't currently have the capacity to provide guidance about this feature or to review a PR. That said, we are interested in adding this at some point, so please feel free to check back about this feature in the future. We'd also be interested to hear from others who would find this feature useful.

Also on the roadmap is a separate (though perhaps related) feature about expanding access to Form Attachments and Submission Attachments. For example, see:

2 Likes