Create and update Entities from repeats

[gareth]

@ahblake

the calculate for ../meta/entity/@id is now blank for create or update in collect and web forms, I tried adding a trigger to it and putting it inside a concat, but it's just blank!

I think this is a typo in your calculation for goal_UUID - when I add another ../ it works as expected, ie: ../../meta/entity/

@ahblake

when created in web forms, so all new UUIDs, the values in the submission matched the values in the entity list for repeat elements 1 & 2, not 3. Trying again, it matched for 1-9 but not #10.

The next thing is there is an unnecessary setvalue which updates the /data/rep_goals/meta/entity/ on odk-new-repeat . In my testing this was setting the uuid for the entity, but nothing was updating the entity_UUID so that’s where they got out of sync for every goal except the first. If I remove this setvalue everything stays in sync.

There is definitely something fishy going on with editing - the selected_at behaviour isn’t at all what I would expect. I’m going to pull out a minimal case just for that…

NB I’m testing in WF v0.18.2

[ahblake]

@gareth

I think this is a typo in your calculation for goal_UUID - when I add another ../ it works as expected, ie: ../../meta/entity/

I went back to the first form, and it's the same expression at the same level (group in a repeat), made another submission just now and the field is populated.
I assumed that I didn't need a ../ for every level in the hierarchy, this could easily have been my misunderstanding, but doesn't explain why it works in one form and not another.

I changed the expression, and the field is now populated in the extract form, but in WF for the update case the value for it only matches the entity list for repeat#1, the values in repeat 2..n are unrelated to the entity list. Collect matches for all. This is also what I was seeing in my original form before the value blanked out.

screenshots

WF, repeat1:

WF, repeat2: 0561e2fa... does not exist in the entire entity list

Collect, repeat1:

Collect, repeat2:

@gareth

If I remove this setvalue everything stays in sync.

sorry, can you explain what you removed / what the unnecessary setvalue is?

Appreciate the help and pointing out my errors!
Also on v0.18.2 here.

[gareth]

@ahblake

sorry, can you explain what you removed / what the unnecessary setvalue is?

Ah, apologies, ignore that. I was deep in the XML that is generated from the xlsx, but that doesn’t help you!

I went back to look at the documentation for adding and updating entities from a form which doesn’t have the repeat but fundamentally is what you’re after. I think the main thing is you’re generating uuids in multiple places which is causing it to get out of sync. Instead you want to make sure the @id is correct for both editing and creating and reference that everywhere you need to.

My changes…

1. In the entities tab, change the entity_id to if (${create_update}='update', ${entity_UUID}, once(uuid())) . This becomes the single source of truth for the entity ID.
2. Then update the entity_UUID calculation to remove the coalesce, ie: selected-at(${calc_ext_goal_IDs} , ${rep_number} - 1). This will be empty when creating, but it should only be used in the @id calculation now.
3. I was still getting the error in test_UUID until I removed the once .

With these changes updating the second repeat I get this…

Let me know if these changes work!

[ahblake]

I originally had the entity_id as a (bit more roundabout) standalone expression, not referring to a form field - I put that in the 'notes' sheet, it was reversed logic but basically the same.

if(${create_update}='create',uuid(),instance('Goals')/root/item[selected-at(${calc_ext_goal_IDs} , ${rep_number} - 1) = name]/name)

I updated it to the following, just to separate it from the form field calculated value (as well as removing the coalesce from the form field to use as a comparison - this did now return blank for the create case);
if (${create_update}='update', selected-at(${calc_ext_goal_IDs} , ${rep_number} - 1) , once(uuid()))

The @id value was still correct for repeat #1, wrong (new UUIDs) for the rest. Deleting eg repeat #2 and re-adding returned yet another new UUID. Deleting all and adding a fresh #1 also returned a new UUID, not the expected value.

I never saw an error in test_UUID - it always showed the expected value from the entity list.

---

To try to get correct behaviour I followed your changes more closely and duplicated the entity_id value

1. entity_id as if (${create_update}='update', ${entity_UUID}, once(uuid()))
2. coalesce was already removed from entity_UUID
3. once removed from test_UUID (field duplicated below to keep it as comparison)

Outcome was unchanged - #1 ok, #2 onwards not ok, delete all repeats and recreate #1 and then #1 not ok. Collect still works for 1, 2, ... and also after deleting all repeats and recreating.

screenshots

#1 original - correct UUID

#2 original - new incorrect UUID

Delete 1 & 2 and recreate, both have new incorrect UUIDs

It seems there's still some other difference between your edit and mine if yours is working properly - but I don't know what it is.

File
webforms entity UUID issue.xlsx (97.5 KB)

[gareth]

Well I think the xlsx is now “correct” as far as I understand it. The problem is when it gets converted into xml there are multiple competing calculations being created. This is what I’m seeing…

Firstly, there’s the bind calculation you’ve defined.

<bind nodeset="/data/rep_goals/meta/entity/@id" readonly="true()" type="string" calculate="if ( ../../../grp_goals/create_update ='update',  ../../../grp_goals/entity_UUID , once(uuid()))"/>

Secondly there’s a setvalue action that is being automatically created that fires when new repeats are added.

<setvalue ref="/data/rep_goals/meta/entity/@id" event="odk-new-repeat" value="uuid()"/>

I think this is causing a race condition - in Collect the former wins, and in WF the latter does. When I comment out that second code block everything works as expected. What I don’t know is, is this a bug in the xls → xform conversion and the setvalue should not be created, OR if the order of operations is incorrect in WF.

@LN Do you have any idea?

[ahblake]

Could your machine/browser also be causing different behaviour to mine if it's a race condition?

This is getting well above my pay grade now - I'm going to watch from the sidelines, but happy to test anything as needed!

Appreciate everything so far @gareth.

[Lindsay_Stevens_Au]

@gareth XForms 1.0 spec s10.1.1 suggests a setvalue may trigger recalculation, in which case it would need run before bind:

XForms Actions that change only the value of an instance node results in setting the flags for recalculate, revalidate, and refresh to true and making no change to the flag for rebuild. The XForms Actions in this category are:

setvalue

Pyxform 4.2.0 outputs a setvalue with uuid(), along with a bind if there is a user-specified entity_id expression, which assumes the bind will overwrite the setvalue in the client. It seems to have been that way since entities could be updated (spec 2023.1.0) with the pattern extended to repeats, and there are tests to check that it does this (for example) - which is not to say it can’t be changed but rather it’s not a random/undefined behaviour.

[LN]

@ahblake thanks for bringing this very interesting case to our attention.

@Lindsay_Stevens_Au

It seems to have been that way since entities could be updated (spec 2023.1.0)

It now seems wrong to me and I think the condition should be just not entity_id_expression -- if there's an entity_id_expression, it should take precedence and be the only way that entity id is set.

That's how the user-facing documentation is written. That is, we say that if you have a form that creates and updates, the entity id expression should account for both by having a path that generates a uuid.

It's true that if setvalue is triggered first things work out but why have the setvalue if we never expect its result to be used? I've filed https://github.com/XLSForm/pyxform/issues/819 but of course if I missed something let me know @Lindsay_Stevens_Au!

@Lindsay_Stevens_Au

it would need run before bind

My understanding from the event sequence is that xforms-model-construct would trigger xforms-recalculate before xforms-ready so for initialization I would expect calculations to run before actions bound to xforms-ready (we've split sxforms-ready into odk-instance-first-load and odk-instance-load). In that sense Web Forms does seem more strictly correct. In general I would not expect the same node's value to be set by both an action and a calculation, though so for now addressing the issue above should be sufficient.

[LN]

ODK Collect v2026.1.0 is now out! It has the full implementation for creating and updating multiple Entities from repeats and groups, including offline. You can create or update as many local Entities in a single submission as your device performance will allow for (most likely limited by expressions in repeats).

We're not officially announcing it yet because XLSForm support is pending, now projected in Central at the end of February or beginning of March, we'll update as we get closer! For the very eager, you can use the repeat column for now, try @Lindsay_Stevens_Au's work in progress at https://github.com/XLSForm/pyxform/pull/818 or modify your own XForms following the spec.

As I wrote above, our current plan is to remove the repeat column on the entities sheet entirely. It was never documented and we're confident only people who follow this thread have used it. If you have an XLSForm with the repeat column, it will fail to convert in the next Central release and you will have to switch it over to the updated format once it's supported. That said, any published form you have based on that column or created by hand will continue working in Collect, Web Forms, etc.

Please let me know if you have any questions.

[Lindsay_Stevens_Au]

@Grzesiek2010 re: Collect only error for relevance from repeat field the recent comments in this thread may be interesting in terms of collect vs webforms behaviour when there are competing setvalue and bind targeting the same node. The intent is for pyxform not to output competing actions like that in future, but still seems like it’s worth noting