Pull submissions starting from date

Ideas
6

I've been studying the code and I have some insights regarding this feature proposal:

  • Briefcase pulls data from Aggregate in batches of a maximum of 100 submissions each.

  • A batch consists on an XML with:

    • A list of submission instanceIDs
    • A "cursor" that can be used to get the next batch

  • A cursor is an XML with:

    • The field used to order submissions. Currently we're using LAST_UPDATE_DATE
    • The last update date (ISO8601) of the last submission from the previous batch
    • The instanceID of the last submission from the previous batch
    • A boolean telling if the cursor is a forward cursor (purpose of this yet to be determined)
    <cursor xmlns="http://www.opendatakit.org/cursor">
  <attributeName>_LAST_UPDATE_DATE</attributeName>
  <attributeValue>2018-11-07T14:43:24.644+0000</attributeValue>
  <uriLastReturnedValue>uuid:d9b67b6f-2058-469b-8cb1-c86b9c34b632</uriLastReturnedValue>
  <isForwardCursor>true</isForwardCursor>
</cursor>

  • In order to get the first batch of submissions, Briefcase sends Aggregate an empty cursor

  • Briefcase will continue asking for more batches until an empty batch arrives, which ends the pull operation

Here's an idea:

  • Briefcase stores the last cursor used for each form
  • We add a checkbox to "resume" the last pull in the Pull tab
  • When that's enabled, Briefcase sends the stored cursor instead of an empty one, effectively resuming the pull operation.

We could even build arbitrary cursors to resume pulls starting from different submissions, but the idea above seems like the smallest possible increment that brings value and would let us test this on the field.