Missing instance error with form that can create or update Entities

Thanks @LN for your support in thread Updating and creating entities with one follow-up form - #2 by LN.

Your solution worked well with the example you provided. I tried to apply this to another survey but fail with the following error message when uploading the xls form on the server:

The entity sheet looks like this:

When removing the content in the columns update_if, create_if and entity_id the upload works fine.

Any idea?

This is a remarkably unfriendly error message, sorry!

It's saying that it's trying to access a __version property from an Entity List named s1e_expoitation_agricole but can't find that list.

How do you have the user pick an existing Entity? Are you maybe using something like a QR code match or something like that? My guess is that you (very reasonably) have assumed that putting s1e_expoitation_agricole as the list name in the entities sheet gives the form access to that list.

In fact, if you're not using a select_one_from_file to select an existing Entity, you need to explicitly attach the list using csv-external. We have some documentation around this here.

If my guess is correct, you'll simply need to add a row to your survey sheet with type csv-external and value s1e_expoitation_agricole.

If that doesn't work, we'll need to know more about your survey sheet. If you can't share the form publicly, feel free to direct message it to me.

Thanks @LN for your reply, which I tried.

Unfortunately I still get an error message when opening the form in Enketo:

The presumed workflow for this form is to:

  1. Check if a given dataset is registered in an external csv (not created from an entity)
  2. If the data is stored in the csv, pull it into the form to check for accuracy and modification if necessary.
  3. If the data is not found, create a new dataset
  4. Store specific fields of the dataset in a entity list
  5. Once all households are registered, pull the data during the next round of datacollection from the entity list (instead the csv), update it or create new datasets to the entity.

s1_exploitation_agricole_test_entities.xlsx (664.0 KB)

Sorry! It's a lengthy form :face_with_spiral_eyes:

Unfortunately I cannot say how, but the error desrcibed in my las post disapeared.

Although the form works fine now, the issues I incounter now are

  1. Entities were originally created, but some fields in the entity (e.g. exploitation_code) were not filled in, while others were (e.g. inspecteur_code). However, all fields are filled in the submissions of the form.

  2. After some changes in the form that I think are not directly linked to the creation of the entities, no more entities are created at all. I have seached in the forum and tried out several options in the create_if and update_if columns of the entities sheet. Without any success.

Thanks for any support,

s1_exploitation_agricole.xlsx (666.6 KB)

I gave it another try.

When leaving empty create_if, update_if and entity_id the entities are created (with all fields populated).

I can't get behind the logic of the behaviour.

I'm sorry you're having trouble with this! Thanks so much for sharing your form and scenario in detail, it provides such great context for digging into and clarifying Entity concepts.

First, I think it's always worth considering whether splitting the create and update actions between two separate forms could work because that makes the form design easier to reason about. I imagine you've considered this but wanted to mention it explicitly.

The downside to using separate create and update forms is there may be quite a bit of redundancy between the two forms and if the form is complex and/or changes frequently, maintaining redundant forms may not be practical.

The other downside is that it means the user has to make a decision about whether they're creating or updating and may get an error message if they pick the wrong one. Given that your form explicitly asks the user whether or not the Entity is already registered, this may not be a significant issue for you. I also notice that to handle the "update or create" for individual properties, you have a pair of fields followed by a calculate so you already have some redundancy. This shouldn't be necessary, you should be able to use a single field corresponding to each property. You would then set the default based on a possible existing value and it would be blank if none existed. In other words, you can use only the "*_existant" fields because they take care of only setting a default if there is one and otherwise leaving the field blank for the user to enter a new value in.

I also notice that you have some calculates on user-editable fields which means that user-provided values will be replaced. Some fields have triggers but not all. I also don't think the triggers are correct -- an update to the default should be triggered by a change in the value that's used to look things up in your list, in this case exploitation_agricole_code (see documentation about this).

I think that here you mean a given Entity. I think using another external csv that is not an Entity List may be making this workflow more complicated than it needs to be. It looks like scopaci_exploitations_agricoles_registre_2023.csv has your existing data and that you may want to update that data. If that sound right, could you pre-populate your entity list (s1e_exploitation_agricole) with that existing data? That would allow you to make updates to it. Otherwise I don't think there are any updates to make -- you're looking at data from scopaci_exploitations_agricoles_registre_2023.csv and there are no corresponding entities in s1e_exploitation_agricole to be updated. I think that's why the create-only case you've now ended up with works as you intended.

update_if and entity_id are needed to configure updates. create_if is optional to configure creates -- if you exclude it you get the same result as if you put true() in create_if. What you've ended up with is a form for which each submission creates a new Entity in the s1e_exploitation_agricole list. Given that your source data comes from another static CSV, that seems appropriate.

Taking a step back -- are you sure you need Entities in this case? Here are some reasons why you might:

  • A household can get multiple updates during the same round
  • A data collection round is not precisely bounded in time so there isn't a set moment for you to download existing data and manually attach it to the form
  • You want households that have been visited this round to be indicated in some way to avoid duplicate visits
  • You want to access updated household information from multiple different forms

You may find this section of the docs helpful.

If none of these apply and you have distinct phases to your work, you may be better served by using a manually-updated CSV. Or maybe you could use Entities with few properties to do things like keep track of which households have been visited during the current round.

Hopefully some of the ideas and questions I've shared help you make progress. I'll also schedule an Entities Office Hour session next week and would be happy to dig more into your scenario if you can join!

Thanks @LN for looking into this troublesome form. This is much appreciated. I take it as part of my learning curve :wink:

I would be more than happy to assist, but will leave on holiday tomorrow and will be out of office until 31. August.

I'll go through your other comments most probably after this date too.

Best, Daniel