Updating and creating entities with one follow-up form

[dast]

Hi,

I am testing entities for our purposes. To this purpose I have created a form that creates an entity list of the enumerators to replace the .csv file we are using at the moment.

This forms works fine and entities are correctly created.

I have build a second form very similar to the first to modify the entities. Works fine as well.

For simplicity reasons I now want to use the same form to update AND create new entities in the same entity list. This fails.

To find the error I have removed in the entities tap the label column. In this case, the form manages to create new entities but cannot modify existing entries. It seems as if updating and creating new entities is not possible with the same form, although this procedure is descibed in the tutorial:

Creating AND Updating Entities with one Form
You can give both a create_if and an update_if rule for the same Form. If only one of these expression evaluates to true or 1, then only that operation will be carried out. If both rules evaluate to true or 1, the Entity will be created if it does not exist (as identified by the entity_id expression), and updated if it does. Of course, if neither rule evaluates to true or 1, no Entity changes will occur.

I attach the xlsform.

Thanks you for any help.

Best, Daniel
admin_inspecteur.xlsx (13.0 KB)

[LN]

Currently your entity_id expression needs to explicitly generate an id in the create case. You can do this with an expression like if(${inspecteur_code_existant='', uuid(), ${inspecteur_code_existant}).

Another way to express this same idea is using the coalesce function: coalesce(${inspecteur_code_existant}, uuid())

I think you were getting at a similar idea with inspecteur_code but that is a user-provided code, not a system ID. You can have both but each entity must have a UUID as its entity_id.

Here's a similar example you may find helpful: https://docs.google.com/spreadsheets/d/1poe2lx_MUzbN0ySuZlRX8_fuKqM4LdsBMwKORQW1QRk I'll work to improve docs around this case.

[dast]

Thanks LN! I'll try your solution as soon as possible and give a reply.

[Devendra_Patankar]

Thanks Great help for same problem.

[dast]

Thanks @LN for your support.

It worked with the example you provided. I now try to apply this to another long survey but fail with the following error message when uploading the xls form on the server:

The entity sheet looks like this:

When removing the content in the columns update_if, create_if and entity_id the upload works fine.

Any idea?

[ahblake]

Boring stuff

I tried multiple ways to write the entity_id expression, and none have worked for the 'create' case, but all work for the 'update' case. ${item_to_update} is the selected entity list item, ${action} is the question to either create or update

Tested in Enketo (update worked, create didn't)

• coalesce(${item_to_update}, uuid())
• if(${item_to_update}!='',${item_to_update},uuid())
• if(${action}='update',${item_to_update},uuid())

Tested in Collect (update worked, create didn't)

• if(${action}='update',${item_to_update},uuid())

And, then I looked at the example linked in this thread and saw two other columns create_if and update_if! These are in the docs, but aren't in the 'trees' example form linked from the docs and the docs don't indicate that you must have them to allow change/update functionality @LN.

So I added ${action}='update' to update_if and ${action}='add' to create_if and reverted to entity_id = coalesce(${item_to_update}, uuid()), works in Enketo for create & update!

For others - the docs aren't explicit - if you want to either create OR update in a single form, you must have create_if and update_if columns in the entities sheet of your form, with expressions that evaluate for those cases. Then the entity_id column must also have an expression to return the ID of the item being updated or generate a UUID if being created. eg

• create_if
  • ${create_or_update}='create'
• update_if
  • ${create_or_update}='update'
• entity_id
  • coalesce(${existing_item},uuid())

[Joshua_Ayanlakin]

Hello!

I have a similar issue with this form I scripted for immunization campaign to track missed children resolved on previous days from e-tally. I found it difficult to load it on ONA, I dont know if it could only load on ODK Central or something is wrong with the form. Please help me. Below is the error code I get upon trying to load the form on ONA:
ODK Validate Errors: >> Something broke the parser. See above for a hint. Error evaluating field 'baseVersion' (${meta}[1]/entity[1]/@baseVersion): The problem was located in Calculate expression for ${entity}/@baseVersion XPath evaluation: Instance referenced by instance(Trees)/root/item/__version does not exist Caused by: org.javarosa.xpath.XPathMissingInstanceException: The problem was located in Calculate expression for ${entity}/@baseVersion XPath evaluation: Instance referenced by instance(Trees)/root/item/__version does not exist ... 10 more The following files failed validation: ${tmphs8ch1xl} Result: Invalid

Your file should have the .xlsx, .xml file extension because Ona uses XLS Forms to create forms.

Not sure what to do? Read more aboutUsing XLSForms

Thank you.

[dast]

Can you share your form?

[Joshua_Ayanlakin]

Dear Dast,

Thank you for your response. Please see attached the form and follow up.

missed children follow-up.xlsx (79.1 KB)
form.xlsx (389.5 KB)

[MinimalPotato]

Hey @dast & @Joshua_Ayanlakin !

Looks like you’ve run into a tricky little bug in ODK Central (yep, this one existed in ODK Aggregate as well). It’s one of those sneaky issues that’s really hard to catch - BUT, I happen to know the fix! I’ll first explain what’s happening and then share the solution.

The Explanation

A few years ago, when I was just getting started with ODK, I spent a lot of time reading the docs, testing things out, and basically trying (and failing) over and over again because that's the best way to learn, right? (For the record, I failed 300+ times while learning about ODK and 10,000+ times while learning to code in general! )

During my early ODK experiments, I ran into quite a few weird bugs - which I ignored as I managed to fix them with quick workarounds or tweaks. One of those bugs popped up when I was exploring the pulldata() and instance() functions.

Now, for many users - these two functions seem to do the same thing: looking up and pulling data from an external file (like a .csv). At first, I was confused too. I asked myself - "If both functions serve the same purpose, then why have two?". My guess was that the developers designed pulldata() to be the simple, intuitive option, while instance() was meant for more advanced filtering and use cases. So, naturally, I decided to experiment with both. That’s when I stumbled upon the very same issue you’re facing now!

What did I find?..

• When you use pulldata() to lookup and fetch data from an external file, it automatically loads the .csv onto the device.
• But when you use instance(), it does NOT automatically load the .csv - which is totally unexpected behavior! Without loading the file, the form simply won’t work.

So, if you're using instance(), you have to manually force the file to load.

Now, here’s the connection to your issue:

• The "update_if" and "create_if" fields in the entities sheet behave just like the instance() function in this case - which means you’ll need to manually trigger ODK to recognize the external file / entity list. But how? Check the next section!

The Solution

To make this work,

Step 1: Add the following row at the top of your survey sheet:

type	name	label	choice_filter	relevant
select_one_from_file EntityListNameHere.csv	anything_here	anything_here	false()	false()

Just replace "EntityListNameHere" with the exact name of your entity in the entities sheet (under list_name).

Step 2: Manually create the entity list in ODK Central with the same name. No need to add any properties - just creating the entity list will do!

Step 3: Since we’re not using select_one_from_file to get input from the user to update the specific entity record (because if we were, we wouldn’t have run into this issue in the first place ), you’ll need to use something else - maybe a text field, a fixed calculate field, or any other method that fits your form’s logic (like maybe you wish data collectors to scan the barcode for record selection rather than selecting from a list). Just make sure you’re capturing the data in a way that allows you to update the entity record using a specific entity_id.

Step 4: Upload the draft, then publish! Once you do that, your form should start working!

Final Note

@dast - This is why @LN ’s example works. It includes select_one_from_file participants.csv, where "participants" happens to be the dataset/entity list name. Try changing it to "select_one_from_file abcd.csv", and you’ll see the same error pop up again!

@Joshua_Ayanlakin - While this fix will solve your entity-related issue above, I have to say… I’d recommend making a few changes and check the logic in the form for better usability. You might need to add uuid() under the entity_id field and introduce a label field, develop further logic, etc.

Hope this helps! Wishing you a great day!

[Joshua_Ayanlakin]

Many thanks for your response. I will try this out and provide feedback on the outcome.

[LN]

I'm sorry @dast, I see I never got back to you in this thread!

@MinimalPotato is right that the (terrible) error message Instance referenced by instance(Trees)/root/item/__version does not exist implies that the Trees Entity List is not attached to the form. If in your case it's appropriate to show Entities in a select, the approach @minimalpotato described with select_one_from_file would be a good one.

It looks like what you had been trying to do is generate an ID within your form and use that to make an update without having the Entity List attached. That's not currently something you can do. If you can change the structure to use a select, that will be your simplest approach. If it makes more sense in your scenario to look up the right Entity to update based on a calculated ID, you could use csv-external to attach the Entity List without a select. I think that you'll find this documentation helpful.

In order to update an Entity, you need to have its system ID. If you want to use an ID that's calculated or scanned from a barcode, you'll need to look up the system ID based on another property's value. The system ID is accessible through the name property (this is to make it easy to build selects on Entity Lists). You will need a calculate that looks something like

instance("trees")/root/item[barcode_id = ${barcode_id}]/name

You will then reference that value in the entity_id column on the entities sheet.

Here's another thread discussing this: Why is my submission with a barcode id as the Entity id failing to update the appropriate Entity? - #2 by LN

[Joshua_Ayanlakin]

Thank you @ LN. I appreciate your intervention. I am trying to review the form and will post the update here once am done.

[loodbirk]

Hi There

I know this is an old thread, but I am totally at whits end regarding updating of entities.

I have been combing the documentation for solutions. I have tried to fix this problem from so many angles it is not even funny. I have even uploaded “fixed XLSForms” to my development area and even then I cannot get entities to update.

My form logic seems sound to me and I can very clearly see that this is a new entry or that I am updating and entry using notes fields to display the statuses of the entity fields that needs changing.

I have made sure that list_name, label, entity_id, create_if and update_if fields effectively is created and populated

I have tried pulling the fields before editing as well as using the instance command to pull data to be edited, but on submission it changes the submission but not the entity data.

I am running ODK Central version as follows:

versions:
56c866fb4a995dee18bda5ce9e1a828e6863c413 (v2025.3.1)
 351f3b92a9e2e2cdbd1e05f2a2137055afa3473b client (v2025.3.1)
 7739129581f690a7fd51eca436d2f37bf544b7a7 server (v2025.3.0)

Everything else is working perfectly. Normal submission, look-ups, API, editing, deleting all are working as per normal. I just cannot get the entities to edit.

Is there somebody out there that can please assist in this matter?

Best regards

ODK XLSForm - Fleet (20251110).xlsx (647.9 KB)

[seewhy]

CAVEAT: I'm not the best fault tracker, so these are just some brief observations so that someone better qualified can help...

You seem to have an external csv, but haven't specified it's structure (obviously the content might be sensitive, so don't upload it!) so there is a bit of guess work.

I can't see a way of selecting an existing entity (and then populating the form with it's existing values - you would use the calculate cell and use the relevant instance formula to find the corresponding value). Usually you might have a select_one with some choice filters...

Rows 54-74 are all calculates that duplicate values to add to the entity - you could just put the corresponding save_to values in the previous rows and remove these. Your submission dataset will have all these extra columns that are superfluous

It feels like there is a circular relationship between f_id and ef_id and they are drawn from the fleet csv? This doesn't feel right - when does the fleet csv get updated? Or is fleet meant to represent your entity list?

I don't think you need to save_to f_id if you are creating entities - that becomes a duplicate column - in your entity_id (on the entity sheet) you are coalescing either ef_id or f_id (but ef_id is a look up of f_id from your fleet csv - as highlighted above).

So maybe there is some confusion in your mind (certainly in mine) about the csv and entities - it looks like you are conflating them?

I hope these are useful pointers - they are not the answer, but if you can solve the issues and come back with a revised form, with some additional context to answer my queries, someone might be able to give you better advice.

It taking me a while to get my head around entities - it is a new workflow compared with csv, the cross-over is useful in moving towards entities but don't expect everything to be the same...

[loodbirk]

Hi there 'seewhy'

Firstly, thank you for responding and my apologies for also responding only now.

I have to say, by the time I added the file to the post it has undergone quite a few changes in the attempt to get the entity to update. So maybe from my side using the comment "I have tried to fix this problem from so many angles" should have been thrashed out more.
In my defense, I wanted to see if this post is still live first.

Let start with what I have tried.

In my first attempt, I had the entity save_to fields in the group that you see in my sheet as "NEW FLEET NUMBER ENTRY" (rows 6 to 27). In that instance, it created the entity properly and I could use the data in the entity in the other forms I have set up. However, on edit, the entity data would not update.

I then read up some more, looked at some examples and implemented the form logic on the sheet rows 76 to 79. What that does is it looks up the fleet number from the entity data (entity_check_fleet_num) and then pulls the EF_ID from the entity data. If the EF_ID and F_ID differs or EF_ID is absent the form logic would say it is a new record and if it is the same it would mark it as an update. That logic I then used in the entities tab as create_if (${update}='new') and update_if (${update})='update'. It does not show that in the attached sheet. Anyhow, that made no difference to the problem at hand.

Then I created a sheet where I could select it is a new or an update right at the top of the form. That then showed one of two groups. One that you enter the new info and it creates the entity data (tested) and one which you would select the fleet number to update and pull all the relevant data into different fields. I then made it that whether it is new or update the correct data would end up in the fields in rows 54 to 74. The create_if and update_if logic was simple for that one. In any case, that did not work either. I could create the entity, but not change the values. This solution also poses another problem because I use the fleet data in a report pulled by ODATA so the main fields then becomes blank in the report.

As to the entity ID, it is a UUID which is create on new and read in from the form and the entity and checked to create the update or new logic. I have tried various combinations in the entity_id field, but with no success

I downloaded the example given by the ODK team in the documentation and this is where I am getting worried, that did not work as advertised as well.
This got me a bit worried as the example should work, but does not.

So. Either I am missing something crucial, or my server has an issue that I cannot detect (it does not give any errors whatsoever that I can see) or the world is truly flat and the turtle with the elephants are swimming upstream.

In your post you mention the form is connected to a csv. Is there another way of accessing/linking the entity data to look up the fleet number? In my form to get the data from the entity I use the instane/xxxxxxxxx command to get the data and not pulldata that you would use for normal CSV file lookup. Could that be the problem? Am I missing something?

Could you maybe help me with a working example I can work from to get an idea?

All and any help or comments are really appreciated.

[seewhy]

Hi @loodbirk
I think I have got your form and entities working... You can create new entities and update existing ones with this single form.

Basically there is a select_one_from_file at the start (the entity list) and everything cascades from that.

I removed the f_id and ef_id and used instead fleet_id - the select_one (which corresponds to the __id in the entity csv / ODATA) - it checks if fleet_id is empty, and if so, shows fleet_new AND create a new entity. The fleet_num is the one that actually stores the fleet number (and corresponds to label).

I also removed the calculate rows 54-74 - they are not required to make this work.

If you want to update an existing Entity, it pulls the record and populates the form (the calculate 'instance' with trigger for each attribute that is in the entity list). Because fleet_id is then not empty, the update_if statement becomes TRUE.

Note the entity_id statement, and that you don't save fleet_id / fleet_new or fleet_num in the save_to column (they are dealt with by the entity creation, but could cause confusion if you are only looking at submissions - so that's why the fleet_num is important to coalesce - the logic prevents being able to have values in both!)

You will need to add some constraints to avoid duplicate fleet number and anything else that might be unique. And you'll need a way of filtering your entity list if it becomes long!

So with some tweaking for usability, this should get you on your way. I haven't checked all your relevant logic, but at least it doesn't break the parser in Central...

Note that I removed all the required statements except one (so that if it's a new vehicle you need to add a fleet number) - I don't like testing with lots of required fields because you can't skip around the form as easily or submit part-filled forms quickly...

ODK XLSForm - Fleet updated.xlsx (641.7 KB)

Good luck

[loodbirk]

Hi there SeeWhy

Thank you for the feedback. Let me have a look at the file and report back.

Once again, I really appreciate your input on this matter.