Hi, Is it possible to get incremental data using the ODK central API?
Please help.
As of Central v1.3
The Submission update timestamp is now included in OData (as
__system/updatedAt). Resources that accept the$filterquery parameter can be filered on__system/updatedAt.
Which I think you could use in a workflow to only pull submission data that is new since your last successful pull?
Can you please provide some more details about your use case any why you need incremental data?
__system/updatedAtfilter will give you incremental data on updates only. You'd have to adjust the filters to track new submissions and deletes via the__system/submissionDateand__system/deletedAt, respectively.
I am curious whether the API intends to support incremental data updates via a cursor... though I understand a change like this would affect many customers.
A solution could be:
When a submission is created, both __system/submissionDate and __system/updatedAt is modified. When a submission is deleted, both the __system/deletedAt and __system/updatedAt is modified. When a submission is updated, the __system/updatedAt is modified. In this case, __system/updatedAt acts as the cursor to track all changes.
Is there another way to get incremental changes? For context, ODK Central is used as a data source for ELT pipelines.
2 Likes
Edited to clarify: This $skiptoken unfortunately isn't useful for your use case, just filtering by the datetime as suggested by @amritaramnauth should generally work.
I see we don't mention it in the Central API change log, but in v2024.3, we introduced an opaque cursor called $skiptoken.
See the OData Endpoints section of the docs:
As of ODK Central v2023.4, @odata.nextLink contains a $skiptoken (an opaque cursor), which allows you to page through Submissions with a consistent offset, even while new Submissions are being created.
We have been using this in Frontend for scrolling and pagination. It just encodes a submission instance ID though.