My organisation recently upgraded our ODK Central and we noticed a warning message that said “If you have 20k or more versions of a single form (generally generated by an automated script that uses the API to update a form attachment)…”.
This doesn't apply to us, but it's a feature we would like to explore. We currently update our preloads (form attachments) by downloading data from our database periodically (every few months unless there is a big change in the dataset), but having an automated script to update a preload every week would be very useful.
The warning copied above suggests this is an established feature, but our team doesn't have the technical expertise at the moment. I would appreciate any advice on how we might achieve this.
Entities (see documentation) is the concept we are building out to automate this kind of update from within Central.
Entities exist at the Central project level so they can be shared between multiple forms if needed. Starting in Central v2022.3, forms can be defined to automatically create Entities. If you only need to append to your form attachments, Entity creation may be enough to replace your manual workflow. You can also edit existing Entities through Central. You can't currently delete Entities through the frontend, this will be part of the first Central release of 2024.
In Central v2023.5 which will be released in the next couple of weeks, you'll be able to define forms that update Entities. This will let you keep status information right on the Entity whose status is changing and have that status be available across forms.
Because Entities are managed separately from forms and submissions, they avoid the issue that the warning you saw was about. For those starting new projects that require these kinds of automatic updates, I would strongly recommend starting with entities rather than form attachment updates (but for existing projects I generally like sticking with what works!).