Create and update Entities from repeats

[LN]

Developers may be interested in the companion ODK XForms spec proposal.

Background

Repeats make it possible to capture data about multiple things of the same kind within the context of a single form. This means capturing the item data can be embedded in a broader flow and there can be logic used to perform validation between those items.

Users want to create Entities from those repeat so that they can follow up with them. Some practical examples:

• In an agricultural context, it's common to have to capture information about many fields in the same farm. See @dast's Insiders lightning talk and showcase post.
• Household surveys often involve capturing info about individual household members associated with that household. See Form linking

Currently, a single form submission can only create or update a single Entity. We propose making it possible for a submission to either create or update a single Entity OR create/update multiple Entities in a single Entity List from a repeat.

XLSForm spec

entities sheet:

We propose adding a repeat column in which a reference to a repeat can be specified. All other references used in other columns such as label, create_if, entity_id, etc, would be evaluated in the context of that repeat.

For now, exactly one Entity declaration would be allowed.

list_name	repeat	label
trees	${tree}	${species}

survey sheet:

When an Entity declaration is specified on the entities sheet, save_to values would only be allowed in that repeat.

type	name	label	save_to
begin_repeat	tree	Tree
geopoint	location	Get tree location	geometry
text	species	Species	species
end_repeat

Future work

In the future, we expect to further expand to the following:

• create/update multiple Entities in a single Entity List from a repeat AND a parent Entity in the same or a different Entity List
• create/update multiple different Entities from the top level and from multiple repeats

Please let us know if you have any feedback!

[chun_hing_yap]

Absolutely in support of this — great initiative!

[dast]

This would be a great step forward. Do I understand correctly that this is not yet implemented and that only one entity from one repeat could be populated with data from the submission?

To bridge the gap until this feature is properly implemented in ODK we have developed a script for pyodk to retrieve selected data from repeats of a given form, save it in a csv which then is uploaded to Central to populate the form the next time the dataset is used. Would that be worth to share in a showcase?

[seewhy]

@dast

To bridge the gap until this feature is properly implemented in ODK

You can also do this with QuODK - you can load the submission data (repeats) into QGIS then use that as the layer to create (or update) an entity list (exported as a CSV). Followed by a pyODK script to talk to Central.

So certainly not as simple as your tool, but there is the opportunity to adjust any geometry [or any other attributes] if needed, so it has some 'strength'. You can also be selective as to which attributes to include in the entity list.

I'd like to say that I'd thought of this specific use case when designing QuODK, but I confess that I only realised it was applicable when I read your post.

@dast

Would that be worth to share in a showcase?

I would find that useful - any 'grown up' scripts for pyODK would be really useful. My kindergarten scripts are a little wobbly so I find examples a good way to learn.

[Thalie]

Hi @LN,

Exciting developments! A small clarification for implementation with this future spec, how would properties shared across all repeat entities be managed? For example, in a household survey you might first interview the household head, then enumerate household members in a repeat. If I want to register the household’s village (selected, say, from a predefined list in the main body of the XLSForm) as a property of the entity shared between all entities for this submission, will it be possible to save_to outside the repeat? Or do I understand correctly that instead I will have create a calculated field inside the repeat that retrieves this value, so it can be stored with save_to?

[dast]

You can also do this with QuODK - you can load the submission data (repeats) into QGIS then use that as the layer to create (or update) an entity list (exported as a CSV). Followed by a pyODK script to talk to Central.

I still have to test your tool. So many things going on at same time

I would find that useful - any 'grown up' scripts for pyODK would be really useful. My kindergarten scripts are a little wobbly so I find examples a good way to learn.

I’m not quite sure if our script really is grown up. AI was helping a lot. But yeah, it seems to work.

[ahblake]

On the assumption that Collect beta functionality exists for entities from repeats, I made a quick test form in a similar manner to @LN's example at the top to see what would happen, for now only with save_to against fields in one repeat as also creating entities outside a repeat is incomplete. Using Central 2025.3.1

Spec is still at 2024.1.0 so I guess this thread is the spec discussion for 2025.1 in the current Collect beta?

I modified an existing demo entity form to put a repeat around the fields and added the repeat column to the entities sheet and found;

• Enketo - allows the first repeat. In the second, third etc repeats the top note shows a read only text field under it and then the following question has no selections visible and the questions after that are not shown at all.
• WF - allows >1 values in the repeat, can create/update ok.
• Collect 2025.4.0 beta 2 with experimental entities spec 2025.1 off: form doesn't load
• Collect 2025.4.0 beta 2 with experimental entities spec 2025.1 on: can create/update multiple entities ok

@Tyler_Depke - looks like we can start testing repeat entities workflows

[ahblake]

I spotted this in the docs, saving the entity ID in the submission that creates/updates it via this as a calculate:
/data/meta/entity/@id

This didn't work as is for entities from repeats, I expect I need to include the repeat name somewhere in the XPath, looking at a submission XML I can see <meta> and @id under the repeat and this works:

/data/your_repeat_name/meta/entity/@id

Will groups affect this for some future implementation where a single submission can create entities from repeats plus outside the repeat, either inside or outside a group that would need to identify where the meta block exists? Currently the group in my repeat is at the same level as the meta block.

[dast]

Finally managed to test this new feature.

All worked as expected and described in the specs by @LN , although I didn’t test yet create_if and update_if.

But yes, It only works with the Beta version of Collect with experimental entities spec 2025.1 on. Thanks to @ahblake. I would not have found this in the Project settings –> Experimental.

But this feature will only reveal its true power once the following are possible at the same time:

• Entities of the parent area

• Entities of one or more repeat groups

We continue with imperfect work-arounds until then.