Starting with Central v2023.5, it has been possible to update Entities from form submissions. As part of that work, we introduced ODK’s model for handling multiple Entity updates based on the same version: all updates are applied in the order that they are received and Central detects and displays conflicts where they can be resolved or dismissed. You can read about this approach in the original proposal and in the documentation.
Conflict detection is based on an incrementing integer “base version” that each Entity update includes. This represents the Entity version that the update is intended to be applied to. In our original proposal, we wrote:
Sequential integer versions are easy to work with but are not guaranteed to be unique across clients and don’t uniquely identify the contents of updates. That means they do not work well in contexts where multiple offline clients are likely to make several updates AND submit those updates at the same time. For example, clients A and B get EntityA version 1. They both go offline and generate versions 2, 3, 4. If they then submit at the same time, the server may interleave versions from the two clients leading to a difficult history to understand. We believe that this kind of scenario is rare.
As we are getting more deeply into the offline entities implementation, we have decided to better handle this case:
- We think it will be somewhat common for a server-based edit to happen while clients are offline which can also lead to confusing history if not detected
- We initially thought we would prioritize assigning Entities to specific users and strongly assume that any given Entity would only be interacted with by one Collect user but we are now hearing of more use cases that involve different individuals interacting with the same Entity
- We would rather have all of the information to make it possible to reconstruct full history even if we don’t display it in Central right away
This is in line with some thinking that we shared last October.
Next steps:
- Post a companion proposal for extensions to the OpenRosa spec
- Schedule a call for anyone interested to discuss these proposals
- Continue exploratory implementations to further validate the proposal
The concept: keep track of offline branches
We propose adding the following concepts:
- Branch ID: a unique identifier (UUID) grouping together submissions that update the same Entity while offline. When a client receives a server update that applies to a specific entity, it generates a new branch ID which gets used in all update submissions until the next server update. This then makes it possible for the server to differentiate between sequences of updates from different clients or made at different times from the same client and to guarantee that submissions within a branch are well-ordered.
- Trunk version: the last Entity version that the client got from the server. This stays the same for every update made offline whereas base version is incremented by 1 with each update. For the first update in a branch, the trunk version and base version are equal. This information will make it possible for Central to detect if a whole branch was based on an older Entity version.
The branch ID would be included in submissions that result in Entity creation or update and the trunk version would be included for Entity updates only.
We propose using a trunk/branch metaphor that will be familiar to users of version control systems. These concepts will not be exposed to end users who will continue to see a linear update history (at least initially).
XForms spec
We propose making the following additions to the XForms Entities update spec:
- MAY have a
branchId
attribute which is populated with a UUID generated by clients with an offline Entity representation.- Clients that do have an offline Entity representation MUST keep the same
branchId
for a given Entity for an entire offline update sequence. They MUST generate a newbranchId
for a new update sequence after receiving a server update. - When a
branchId
attribute is present in a submission, servers SHOULD attempt to process submissions with that same attribute in the correct order - When a
branchId
attribute is present, atrunkVersion
representing the latest version received from the server MUST also be present. - When a server receives values for
branchId
andtrunkVersion
, it must send those values back to clients as__branchId
and__trunkVersion
when clients get updates for that Entity.
- Clients that do have an offline Entity representation MUST keep the same
Adding these new concepts will result in an ODK XForms specification version update as described here. This means older versions of Collect and Central without offline Entities awareness will not accept forms with this new functionality. Clients like Enketo that are not Entity-aware will continue to work as they do now (without an offline representation of Entities).
These changes will be introduced as Entities spec version 2024.1.0. Clients should only apply creates and updates offline for forms with that spec version or higher.
XForms example
<model odk:xforms-version="1.0.0" entities:entities-version="2024.1.0">
<instance>
<data id="mysurvey" orx:version="1">
<meta>
<instanceID/>
<entity dataset="trees" id="" update="" baseVersion="" trunkVersion="" branchId="" />
</meta>
</data>
</instance>
...
<bind nodeset="/data/meta/entity/@id" type="string" calculate="/data/tree" />
<bind nodeset="/data/meta/entity/@update" type="string" calculate="true()"
<bind nodeset="/data/meta/entity/@baseVersion" type="string" calculate="instance('trees')/root/item[name=/data/tree]/__version" />
<bind nodeset="/data/meta/entity/@trunkVersion" type="string" calculate="instance('trees')/root/item[name=/data/tree]/__trunkVersion" />
<bind nodeset="/data/meta/entity/@branchId" type="string" calculate="instance('trees')/root/item[name=/data/tree]/__branchId" />
</model>
The XForm for an Entity create would not include any of branchId
, baseVersion
or trunkVersion
.
No longer planned -- XLSForm: Explicitly opt out of offline Entities
Given the XForms spec proposal above, it's possible to opt out of offline Entity support by omitting the branchId
attribute. We could expose this capability in XLSForm. This can be helpful if:
- We’re not confident all users will be able to update to a version of a client with offline Entity support
- We know we’re going to be online most of the time and don’t need clients to store state offline
- We’re using a mix of clients, some of which may not have offline Entities, and we want them all to behave the same way
We're not sure how significant these considerations are so are currently leaning towards NOT exposing this. One of our goals in sharing this proposal is to get feedback on whether this seems important to do.
If it doesn't serve immediate user need, we would rather omit it because each configuration point adds complexity and opportunity for bugs to be introduced.
If we do end up wanting to make this configurable, here is our proposal:
We propose adding this as an XLSForm setting on the entities
sheet. This means that when we support creating or updating multiple Entities per form, whether creates and updates are applied offline would be controlled independently for each affected Entity List (and each form since this is a form-based setting).
We propose introducing an offline
column to the entities sheet with allowed values ‘yes’ and ‘no.’ When the value is ‘no,’ the trunkVersion
and branchId
attributes will be omitted in the converted XForm. For now, the entities-version
attribute will be set to 2023.1.0
for compatibility with older clients.
Entities sheet
list_name | entity_id | offline |
---|---|---|
trees | ${tree} | no |
XForm
<model odk:xforms-version="1.0.0" entities:entities-version="2023.1.0">
<instance>
<data id="mysurvey" orx:version="1">
<meta>
<instanceID/>
<entity dataset="trees" id="" update="" baseVersion="" />
</meta>
</data>
</instance>
...
<bind nodeset="/data/meta/entity/@id" type="string" calculate="/data/tree" />
<bind nodeset="/data/meta/entity/@update" type="string" calculate="true()"
<bind nodeset="/data/meta/entity/@baseVersion" type="string" calculate="instance('trees')/root/item[name=/data/tree]/__version" />
</model>