Form spec proposal: support Entity updates from multiple offline clients

Starting with Central v2023.5, it has been possible to update Entities from form submissions. As part of that work, we introduced ODK’s model for handling multiple Entity updates based on the same version: all updates are applied in the order that they are received and Central detects and displays conflicts where they can be resolved or dismissed. You can read about this approach in the original proposal and in the documentation.

Conflict detection is based on an incrementing integer “base version” that each Entity update includes. This represents the Entity version that the update is intended to be applied to. In our original proposal, we wrote:

Sequential integer versions are easy to work with but are not guaranteed to be unique across clients and don’t uniquely identify the contents of updates. That means they do not work well in contexts where multiple offline clients are likely to make several updates AND submit those updates at the same time. For example, clients A and B get EntityA version 1. They both go offline and generate versions 2, 3, 4. If they then submit at the same time, the server may interleave versions from the two clients leading to a difficult history to understand. We believe that this kind of scenario is rare.

As we are getting more deeply into the offline entities implementation, we have decided to better handle this case:

  • We think it will be somewhat common for a server-based edit to happen while clients are offline which can also lead to confusing history if not detected
  • We initially thought we would prioritize assigning Entities to specific users and strongly assume that any given Entity would only be interacted with by one Collect user but we are now hearing of more use cases that involve different individuals interacting with the same Entity
  • We would rather have all of the information to make it possible to reconstruct full history even if we don’t display it in Central right away

This is in line with some thinking that we shared last October.

:zap: Next steps:

  • Post a companion proposal for extensions to the OpenRosa spec
  • Schedule a call for anyone interested to discuss these proposals
  • Continue exploratory implementations to further validate the proposal

The concept: keep track of offline branches

We propose adding the following concepts:

  • Branch ID: a unique identifier (UUID) grouping together submissions that create or update the same Entity while offline. When a client receives a server update that applies to a specific entity, it generates a new branch ID which gets used in all create and updates submissions until the next server update. This then makes it possible for the server to differentiate between sequences of updates from different clients or made at different times from the same client and to guarantee that submissions within a branch are well-ordered.
  • Trunk version: the last Entity version that the client got from the server. This stays the same for every update made offline whereas base version is incremented by 1 with each update. For the first update in a branch, the trunk version and base version are equal. This information will make it possible for Central to detect if a whole branch was based on an older Entity version.

The branch ID would be included in submissions that result in Entity creation or update and the trunk version would be included for Entity updates only.

We propose using a trunk/branch metaphor that will be familiar to users of version control systems. These concepts will not be exposed to end users who will continue to see a linear update history (at least initially).

XForms spec

We propose making the following additions to the XForms Entities creation spec:

  • MAY have a branchId attribute which is populated with a UUID generated by clients with an offline Entity representation.
    • When a branchId attribute is not present, clients MUST NOT make created Entities available offline.
    • When a branchId attribute is present in a form definition, clients SHOULD make created Entities available offline. Clients that do have an offline Entity representation MUST keep the same branchId for a given Entity for an entire offline create/update sequence. They MUST generate a new branchId for a new update sequence after receiving a server update.
    • When a branchId attribute is present in a submission, clients SHOULD attempt to process submissions with that same attribute in the correct order.

We propose making the following additions to the XForms Entities update spec:

  • MAY have a branchId attribute as described in Entity creation above.
    • When a branchId attribute is present, a trunkVersion representing the latest version received from the server MUST also be present.

XForms example

<model odk:xforms-version="1.0.0" entities:entities-version="2024.1.0">
  <instance>
	<data id="mysurvey" orx:version="1">
      <meta>
        <instanceID/>
        <entity dataset="trees" id="" update="" baseVersion="" trunkVersion="" branchId="" />
      </meta>
	</data>
  </instance>
  ...
  <bind nodeset="/data/meta/entity/@id" type="string" calculate="/data/tree" />
  <bind nodeset="/data/meta/entity/@update" type="string" calculate="true()"
  <bind nodeset="/data/meta/entity/@baseVersion" type="string" calculate="instance('trees')/root/item[name=/data/tree]/__version" />
  <bind nodeset="/data/meta/entity/@trunkVersion" type="string" calculate="instance('trees')/root/item[name=/data/tree]/__trunkVersion" />
  <bind nodeset="/data/meta/entity/@branchId" type="string" calculate="instance('trees')/root/item[name=/data/tree]/__branchId" />
</model>

The XForm for an Entity create would include the branchId attribute but no baseVersion or trunkVersion.

XLSForm: Explicitly opt out of offline Entities

Adding these new concepts will result in an ODK XForms specification version update as described here. This means older versions of Collect and Central without offline Entities awareness will not accept forms with this new functionality. Clients like Enketo that are not Entity-aware will continue to work as they do now (without an offline representation of Entities).

Given the XForms spec proposal above, it's possible to opt out of offline Entity support by omitting the branchId attribute. We could expose this capability in XLSForm. This can be helpful if:

  • We’re not confident all users will be able to update to a version of a client with offline Entity support
  • We know we’re going to be online most of the time and don’t need clients to store state offline
  • We’re using a mix of clients, some of which may not have offline Entities, and we want them all to behave the same way

We're not sure how significant these considerations are so are currently leaning towards NOT exposing this. One of our goals in sharing this proposal is to get feedback on whether this seems important to do.

If it doesn't serve immediate user need, we would rather omit it because each configuration point adds complexity and opportunity for bugs to be introduced.

If we do end up wanting to make this configurable, here is our proposal:

We propose adding this as an XLSForm setting on the entities sheet. This means that when we support creating or updating multiple Entities per form, whether creates and updates are applied offline would be controlled independently for each affected Entity List (and each form since this is a form-based setting).

We propose introducing an offline column to the entities sheet with allowed values ‘yes’ and ‘no.’ When the value is ‘no,’ the trunkVersion and branchId attributes will be omitted in the converted XForm. For now, the entities-version attribute will be set to 2023.1.0 for compatibility with older clients.

Entities sheet

list_name entity_id offline
trees ${tree} no

XForm

<model odk:xforms-version="1.0.0" entities:entities-version="2023.1.0">
  <instance>
	<data id="mysurvey" orx:version="1">
      <meta>
        <instanceID/>
        <entity dataset="trees" id="" update="" baseVersion="" />
      </meta>
	</data>
  </instance>
  ...
  <bind nodeset="/data/meta/entity/@id" type="string" calculate="/data/tree" />
  <bind nodeset="/data/meta/entity/@update" type="string" calculate="true()"
  <bind nodeset="/data/meta/entity/@baseVersion" type="string" calculate="instance('trees')/root/item[name=/data/tree]/__version" />
</model>
1 Like