Create and update Entities from repeats

[gareth]

@ahblake

the calculate for ../meta/entity/@id is now blank for create or update in collect and web forms, I tried adding a trigger to it and putting it inside a concat, but it's just blank!

I think this is a typo in your calculation for goal_UUID - when I add another ../ it works as expected, ie: ../../meta/entity/

@ahblake

when created in web forms, so all new UUIDs, the values in the submission matched the values in the entity list for repeat elements 1 & 2, not 3. Trying again, it matched for 1-9 but not #10.

The next thing is there is an unnecessary setvalue which updates the /data/rep_goals/meta/entity/ on odk-new-repeat . In my testing this was setting the uuid for the entity, but nothing was updating the entity_UUID so that’s where they got out of sync for every goal except the first. If I remove this setvalue everything stays in sync.

There is definitely something fishy going on with editing - the selected_at behaviour isn’t at all what I would expect. I’m going to pull out a minimal case just for that…

NB I’m testing in WF v0.18.2

[ahblake]

@gareth

I think this is a typo in your calculation for goal_UUID - when I add another ../ it works as expected, ie: ../../meta/entity/

I went back to the first form, and it's the same expression at the same level (group in a repeat), made another submission just now and the field is populated.
I assumed that I didn't need a ../ for every level in the hierarchy, this could easily have been my misunderstanding, but doesn't explain why it works in one form and not another.

I changed the expression, and the field is now populated in the extract form, but in WF for the update case the value for it only matches the entity list for repeat#1, the values in repeat 2..n are unrelated to the entity list. Collect matches for all. This is also what I was seeing in my original form before the value blanked out.

screenshots

WF, repeat1:

WF, repeat2: 0561e2fa... does not exist in the entire entity list

Collect, repeat1:

Collect, repeat2:

@gareth

If I remove this setvalue everything stays in sync.

sorry, can you explain what you removed / what the unnecessary setvalue is?

Appreciate the help and pointing out my errors!
Also on v0.18.2 here.

[gareth]

@ahblake

sorry, can you explain what you removed / what the unnecessary setvalue is?

Ah, apologies, ignore that. I was deep in the XML that is generated from the xlsx, but that doesn’t help you!

I went back to look at the documentation for adding and updating entities from a form which doesn’t have the repeat but fundamentally is what you’re after. I think the main thing is you’re generating uuids in multiple places which is causing it to get out of sync. Instead you want to make sure the @id is correct for both editing and creating and reference that everywhere you need to.

My changes…

1. In the entities tab, change the entity_id to if (${create_update}='update', ${entity_UUID}, once(uuid())) . This becomes the single source of truth for the entity ID.
2. Then update the entity_UUID calculation to remove the coalesce, ie: selected-at(${calc_ext_goal_IDs} , ${rep_number} - 1). This will be empty when creating, but it should only be used in the @id calculation now.
3. I was still getting the error in test_UUID until I removed the once .

With these changes updating the second repeat I get this…

Let me know if these changes work!

[ahblake]

I originally had the entity_id as a (bit more roundabout) standalone expression, not referring to a form field - I put that in the 'notes' sheet, it was reversed logic but basically the same.

if(${create_update}='create',uuid(),instance('Goals')/root/item[selected-at(${calc_ext_goal_IDs} , ${rep_number} - 1) = name]/name)

I updated it to the following, just to separate it from the form field calculated value (as well as removing the coalesce from the form field to use as a comparison - this did now return blank for the create case);
if (${create_update}='update', selected-at(${calc_ext_goal_IDs} , ${rep_number} - 1) , once(uuid()))

The @id value was still correct for repeat #1, wrong (new UUIDs) for the rest. Deleting eg repeat #2 and re-adding returned yet another new UUID. Deleting all and adding a fresh #1 also returned a new UUID, not the expected value.

I never saw an error in test_UUID - it always showed the expected value from the entity list.

---

To try to get correct behaviour I followed your changes more closely and duplicated the entity_id value

1. entity_id as if (${create_update}='update', ${entity_UUID}, once(uuid()))
2. coalesce was already removed from entity_UUID
3. once removed from test_UUID (field duplicated below to keep it as comparison)

Outcome was unchanged - #1 ok, #2 onwards not ok, delete all repeats and recreate #1 and then #1 not ok. Collect still works for 1, 2, ... and also after deleting all repeats and recreating.

screenshots

#1 original - correct UUID

#2 original - new incorrect UUID

Delete 1 & 2 and recreate, both have new incorrect UUIDs

It seems there's still some other difference between your edit and mine if yours is working properly - but I don't know what it is.

File
webforms entity UUID issue.xlsx (97.5 KB)

[gareth]

Well I think the xlsx is now “correct” as far as I understand it. The problem is when it gets converted into xml there are multiple competing calculations being created. This is what I’m seeing…

Firstly, there’s the bind calculation you’ve defined.

<bind nodeset="/data/rep_goals/meta/entity/@id" readonly="true()" type="string" calculate="if ( ../../../grp_goals/create_update ='update',  ../../../grp_goals/entity_UUID , once(uuid()))"/>

Secondly there’s a setvalue action that is being automatically created that fires when new repeats are added.

<setvalue ref="/data/rep_goals/meta/entity/@id" event="odk-new-repeat" value="uuid()"/>

I think this is causing a race condition - in Collect the former wins, and in WF the latter does. When I comment out that second code block everything works as expected. What I don’t know is, is this a bug in the xls → xform conversion and the setvalue should not be created, OR if the order of operations is incorrect in WF.

@LN Do you have any idea?

[ahblake]

Could your machine/browser also be causing different behaviour to mine if it's a race condition?

This is getting well above my pay grade now - I'm going to watch from the sidelines, but happy to test anything as needed!

Appreciate everything so far @gareth.

[Lindsay_Stevens_Au]

@gareth XForms 1.0 spec s10.1.1 suggests a setvalue may trigger recalculation, in which case it would need run before bind:

XForms Actions that change only the value of an instance node results in setting the flags for recalculate, revalidate, and refresh to true and making no change to the flag for rebuild. The XForms Actions in this category are:

setvalue

Pyxform 4.2.0 outputs a setvalue with uuid(), along with a bind if there is a user-specified entity_id expression, which assumes the bind will overwrite the setvalue in the client. It seems to have been that way since entities could be updated (spec 2023.1.0) with the pattern extended to repeats, and there are tests to check that it does this (for example) - which is not to say it can’t be changed but rather it’s not a random/undefined behaviour.

[LN]

@ahblake thanks for bringing this very interesting case to our attention.

@Lindsay_Stevens_Au

It seems to have been that way since entities could be updated (spec 2023.1.0)

It now seems wrong to me and I think the condition should be just not entity_id_expression -- if there's an entity_id_expression, it should take precedence and be the only way that entity id is set.

That's how the user-facing documentation is written. That is, we say that if you have a form that creates and updates, the entity id expression should account for both by having a path that generates a uuid.

It's true that if setvalue is triggered first things work out but why have the setvalue if we never expect its result to be used? I've filed https://github.com/XLSForm/pyxform/issues/819 but of course if I missed something let me know @Lindsay_Stevens_Au!

@Lindsay_Stevens_Au

it would need run before bind

My understanding from the event sequence is that xforms-model-construct would trigger xforms-recalculate before xforms-ready so for initialization I would expect calculations to run before actions bound to xforms-ready (we've split sxforms-ready into odk-instance-first-load and odk-instance-load). In that sense Web Forms does seem more strictly correct. In general I would not expect the same node's value to be set by both an action and a calculation, though so for now addressing the issue above should be sufficient.

[LN]

ODK Collect v2026.1.0 is now out! It has the full implementation for creating and updating multiple Entities from repeats and groups, including offline. You can create or update as many local Entities in a single submission as your device performance will allow for (most likely limited by expressions in repeats).

We're not officially announcing it yet because XLSForm support is pending, now projected in Central at the end of February or beginning of March, we'll update as we get closer! For the very eager, you can use the repeat column for now, try @Lindsay_Stevens_Au's work in progress at https://github.com/XLSForm/pyxform/pull/818 or modify your own XForms following the spec.

As I wrote above, our current plan is to remove the repeat column on the entities sheet entirely. It was never documented and we're confident only people who follow this thread have used it. If you have an XLSForm with the repeat column, it will fail to convert in the next Central release and you will have to switch it over to the updated format once it's supported. That said, any published form you have based on that column or created by hand will continue working in Collect, Web Forms, etc.

Please let me know if you have any questions.

[Lindsay_Stevens_Au]

@Grzesiek2010 re: Collect only error for relevance from repeat field the recent comments in this thread may be interesting in terms of collect vs webforms behaviour when there are competing setvalue and bind targeting the same node. The intent is for pyxform not to output competing actions like that in future, but still seems like it’s worth noting

[LN]

We have now merged in XLSForm support for multiple Entities from repeats and groups and it is available at https://getodk.org/xlsform/. We also have documentation staged at https://github.com/getodk/docs/pull/2028/changes. This means you can write XLSForms, convert them on the website and then upload the resulting XML in Central to try it in Collect, Web Forms, etc. Unless we find any new issues we expect to release by the end of the week!

Here are some small example forms to give you a sense of what's possible:

• nested_repeat.xlsx (10.2 KB): person and favorite pet at the top level, trees repeat with nested improvements repeat
• CreateUpdateMultipleDatasetsTestForm.xlsx (459.0 KB): create or update form with people and trees

[ahblake]

@LN

Unless we find any new issues we expect to release by the end of the week!

excitement intensifies

poring over the forms this week so I'm ready to try it out!

[chun_hing_yap]

will this feature also work for nested group within repeat such as

begin_repeat
begin_group
end_group
begin_group
end_group
end repeat

[LN]

Can't wait to see everyone's forms!

@chun_hing_yap yes, any group or repeat can define an Entity action so you can do some pretty wild things. There may still be some unhandled edge cases so please let us know if you run into anything unexpected. For example I just identified https://github.com/XLSForm/pyxform/issues/822

[LN]

Central v2025.4.3 is now out and includes XLSForm support for declaring multiple Entities per form definition! Basic documentation is available here and we intend to share a more in-depth tutorial soon.

And remember, with great power comes great responsibility. Please let us know how you use this functionality and if you run into any issues.

[ahblake]

@LN

Basic documentation is available here

I'm sure the docs are currently in a state of flux with this recent release, so this maybe already in hand, but a few suggested areas for edits;

The note at the end of this section is no longer necessary;

Currently, the entities sheet can only have one row because each submission can only create or update a single Entity.

This section could benefit from examples of how to use ../.. for entities within groups/repeats

I think the multiple entity lists part is a little unclear as groups are not mentioned, particularly this sentence;

That is, the only way to create or update multiple Entities in the same list is to use a repeat.

I think it needs to point out that each entity list can only have save_to values isolated to separate groups or repeats, and for multiple entities it (somewhat obviously after reading the section above) must be a repeat. And that groups/repeats can be nested (like the example form).

[Thalie]

I would like to kindly echo the comment from @ahblake about the need for clearer documentation around entity handling in the latest ODK upgrade, particularly regarding how save_to behaves when spread across groups (and not just repeats) in single-entity forms with ODK Central v2025.4.3.

I have just encountered an issue I was not expecting while updating a single-entity form for an upcoming data collection launch. The update was unrelated to entities (adding a conditional filter for 2 variables), but it resulted in the loss of 10 out of 13 entity property links because the save_to fields were distributed across different groups. This came as a surprise, also as the pipeline (which depends on these properties) had been thoroughly tested and working for months, so I was not expecting a change of behaviour and would definitely not have the pipeline if I had simply corrected a typo. While it was very straightforward to consolidate all save_to fields into a single group, I could have easily missed this if I had not checked the property page, so I agree a few examples would be incredibly helpful for others in similar situations.

On a positive note, I am excited about the new multiple entity support, and it has actually prompted me to rethink my current workflow, so I am now testing the addition of a second entity to my form to streamline the data collection. I did not expect to make the leap this soon, but it’s a welcome shift!

[ahblake]

@Thalie

I have just encountered an issue I was not expecting

Did you see this thread?

When using groups, entity fields only processing in first group, ignored elsewhere

We've made the container of an Entity declaration implicit in XLSForm which makes things simple for simple forms but may make reasoning about more complex structures harder. First we need to fix the regression and then we'll keep thinking about what documentation or improvements we can provide to make this easier to reason about. Your example @MinimalPotato has this structure: container1 field1 container2 field2 save to entityList1 container3 field3 save to entityList2 You want …

It sounds like you have a workaround, but your form should work as it was once the issue is resolved.

[Thalie]

Ah yes, sorry I was in hurry as data collection starting today for the form that broke, so I must admit I focused on finding a quick fix and jumped on the first related topic I found . Thanks for pointing to the link!

[LN]

My apologies, I should have cross-linked here! It's a serious regression and I'm really glad you noticed and were able to work around it.

To others in this thread, the issue is that when save_tos are spread across groups in forms that create/update single Entities, some save_tos may be ignored. Forms that have already been published will continue to work as expected but the issue will be revealed if making a form update. As @Thalie discovered, a temporary workaround is to put all save_tos in the same group. We hope to have a patch very soon.