When using groups, entity fields only processing in first group, ignored elsewhere

1. What is the issue? Please be detailed.

Using self-hosted version 2025.4.3.

When using begin_group / end_group blocks with entities, only the first group is able to process entity fields; all other entity fields are ignored. No issue when groups are not used.

2. What steps can we take to reproduce this issue?

  1. Create a very simple test:

In survey sheet:

type name label save_to
begin_group group1
text field1 field1 field1
text field2 field2 field2
end_group group1
begin_group group2
text field3 field3 field3
text field4 field4 field4
end_group group2

In Entities sheet

list_name label
my_dataset ${field1}
  1. Import form. Confirm that only field1 and field2 are processed

  1. Remove groups in survey sheets as below
type name label save_to
text field1 field1 field1
text field2 field2 field2
text field3 field3 field3
text field4 field4 field4
  1. Reimport form. Confirm that all 4 fields are processed

[I cannot embed the screen shot as a new user]

3. What have you tried to fix the issue?

I have created a simple test as above to confirm it is not related to my more complex form. Not much else.

4. Upload any forms or screenshots you can share publicly below.

See attached

my_dataset_with_groups.xlsx (8.7 KB)

1 Like

@LN is this related to 823?

Modifying the attached form to nest group2 inside group1 results in the opposite of the initial problem (and same as issue 823 describes) now only the group2 fields are being recognised

And nesting a 3rd group inside group2 then results in group3 fields being the ones recognised. It is only recognising fields in the first of the deepest level groups.

Edit: As a check I grabbed a form that has save_to values in different groups (1x save_to in 4x different groups), old submissions updated these successfully, reuploading this now as a new form in a different project showed that it would update 1x list with 1x property. Not 4x properties as expected.

The existing published version (Oct 2025) still works as expected in Central 2025.4.3 today, updating all 4x fields.

versions:
b6f7b74e0fad0cbfad7ae507192678ff4e97fd36 (v2025.4.3)
bd9c322760f84f38d323d76072ddf2b74574c102 client (v2025.4.3)
d05e1993552b97040707a3ad912d312af87c0a0b server (v2025.4.2)

2 Likes

I'm not entirely sure whether this is an intended feature or a bug, but I'm really loving the new entity management model introduced in ODK Central v2025.4.3..! :smile: :partying_face:

From what I understand so far:

  • 1 group in the XLSForm = interacts with only 1 entity list.

With this behavior in place, it becomes possible to design some very powerful and automated workflows. For cases involving nested groups, or situations where we need to bring data from multiple groups into a single entity list (the main problem of this forum thread), I've been using the following approach:

type name label calculation save_to
begin_group group1
text field1 field1
text field2 field2
end_group group1
begin_group group2
text field3 field3
text field4 field4
end_group group2
begin_group group3
calculate c_field1 ${field1} field1
calculate c_field2 ${field2} field2
calculate c_field3 ${field3} field3
calculate c_field4 ${field4} field4
end_group group3
1 Like

My understanding was that this only applies if you have multiple entity lists listed, no restriction is mentioned in the docs (and before this release you could definitely place fields across groups).

But, as you say, if this is a bug, unintended consequence, feature! then calculates will be needed to collate them into one group (or at the top /data/ level)

2 Likes

This is indeed an unintended regression, thank you for reporting it @dawansv and to everyone else for chiming in. Our intent is for all forms that worked pre-Central v2025.4.3 to continue working exactly the same way. I think we missed this issue because all our test cases had at least one saveto at the top-level. This form structure makes perfect sense, though, and should be supported.

Sort of! Up to one Entity declaration is allowed per container (form top-level, group or repeat). Savetos correspond to the declaration that is in their nearest parent. That means for the form @dawansv shared, the declaration should be at the top-level and then all the savetos in groups would correspond to that declaration because it's the only one.

In XLSForm you don't explicitly declare which container the declaration goes in, pyxform does a bunch of work to compute that. What we're discovering here is that it's not always getting it right. You're absolutely right that you can work around this unintended limitation with calculates but we consider this broken.

1 Like

@LN You say “I think we missed this issue because all our test cases had at least one saveto at the top-level.” but as far as I can see I have the same problem with the following. It’s only processing field2 and field3 in group1.

type name label save_to
text field1 field1 field1
begin_group group1
text field2 field2 field2
text field3 field3 field3
end_group group1
begin_group group2
text field4 field4 field4
text field5 field5 field5
end_group group2

Ah, I see. Thank you for clarifying @LN !

Actually, the following scenario/design made me think that each container (whether it is the form top-level, a group, or a repeat) would now need to interact with its own entity list, and that these entity lists would strictly remain independent from one another.

For example, suppose I have an XLSForm structured like this:

type name label calculation save_to
text unique_id Enter Unique ID
begin_group general_info General
calculate unique_id_1 ${unique_id} common_info#unique_id
text person_name Enter name of person common_info#person_name
integer person_age Enter age of person common_info#person_age
select_one gender person_gender Select gender of person common_info#person_gender
end_group general_info General
begin_group meal_info Meal
calculate unique_id_2 ${unique_id} meals#unique_id
text breakfast What you had in breakfast? meals#breakfast
text lunch What you had in lunch? meals#lunch
text dinner What you had in dinner? meals#dinner
end_group meal_info Meal
end_group group3

In this case, my intention was to store the person information and the meal information in two separate tables (entity lists). Both entity lists would need to store the Unique ID, so that I can later merge them during analysis. However, the form itself only contains a single unique_id field.

Because of this, I was not sure whether something like the following would work.. :sweat_smile: :

type name label calculation save_to
text unique_id Enter Unique ID common_info#unique_id, meals#unique_id

So I initially thought that keeping the entity lists separated in this way (for clarity) was actually an intended feature, my bad!

We've made the container of an Entity declaration implicit in XLSForm which makes things simple for simple forms but may make reasoning about more complex structures harder. First we need to fix the regression and then we'll keep thinking about what documentation or improvements we can provide to make this easier to reason about.

Your example @MinimalPotato has this structure:

container1
  field1
  container2
    field2 save to entityList1
  container3
    field3 save to entityList2

You want to save field1's value to both lists. There's no way to express saving a single field's value to multiple lists so you need to "forward" field1 with calculates to at least one of container2 and container3. I think forwarding to both as you have done is the easiest to understand.

Technically, per our spec, it should be possible to have an Entity declaration at the container1 level to save field1 and field2 to entityList1 and then only forward field1 into container3 and have the entityList2 declaration in container3. That feels really confusing to me so I would not recommend it. The issues discovered in this thread suggest it probably won't work as it should currently.

We'll need to develop and write up some best practices as we all write more forms with it!

that’s also what I ended up doing to restore the expected behavior (apologies for missing this thread earlier and posting somewhere else; I was in emergency mode… so not thinking very clearly). In my form, only the save_to fields located in the most nested group (root > group 1 > group 2) at the end of the form remained linked to my single entity, while earlier save_to normally were ignored. Something which is not clear for me (as I do not think I have a form left to test this with) and that we discussed with @aurdipas: was this behaviour change introduced at the time of the form update, or would it have affected the already published form as soon as I upgraded the ODK Central server to 2025.4.3?

No, I've tested this, the existing published version that was uploaded pre 2025.4.3 still functions as expected in 2025.4.3, but uploading the identical xlsx in 2025.4.3 will result in the aberrant behaviour.

(apart from collating all your entity save_tos into a group, if you didn't want to change the form fields presumably you could also download the older XML and modify it to add your form edit and update the version then that ought to be ok - pure conjecture)

2 Likes

A post was split to a new topic: Entity draft shown with form title rather than instance name

We have just released ODK Central v2025.4.4 which should fix the originally-reported issue.

Really appreciate the helpful report @dawansv! Please introduce yourself when you get a chance, I'd love to know more about what you're working on.