When editing a submission that created an entity, my uniqueness check says that entity was already created so I can't edit submitted data

Support
1

1. What is the issue? Please be detailed.
I have a form that uses entities to prevent data collectors from entering data from a single observation multiple times. However, this is creating a problem when I attempt to edit entries, e.g., if there are errors. When I open the entry for editing, instead of seeing the data as originally entered, I see the message given when an entry is created for an observation that is already in the system.

2. What steps can we take to reproduce this issue?
As far as I can tell, anyone using entities to prevent double entries of data will encounter this same problem. You can reproduce it by uploading the example form I attach below.

3. What have you tried to fix the issue?

4. Upload any forms or screenshots you can share publicly below.

edit_prob_example_20240109.xlsx (17.9 KB)

1 Like
2

Hi @kristen.jafflin,

Yes, let's figure out your issue!

Thanks for sharing this example form. After playing with the example form, I think I understand what you are trying to do, but I wanted to double-check.

  1. If a data collector tries to enter data, check (based on an ID) if it already exists and stop them from re-entering data
  2. But also show the existing data associated with that ID and let them change it if it is wrong

So everything will happen through this one single form, but the form needs to be able to create an entity, check against existing entities, and also update an existing entity, yes? I can see that the form doesn't quite work to do part of updating an entity.

Could you tell me more about what you mean by the following?

When I open the entry for editing, instead of seeing the data as originally entered, I see the message given when an entry is created for an observation that is already in the system.

What is also probably adding confusion is the different things (submissions, entities) that can be edited and what does or doesn't have an effect on something else. When you 'open the entry for editing', are you:

a) editing the submission that first made the entity (editing a submission after it creates an entity doesn't affect the entity)
b) opening an entity edit pop-up in Central to edit the data of the entity (this is one way for an admin/manager in Central to fix entity data as they see it without a custom form)
c) opening the form to make a new submission, entering an ID to select an existing entry, and then trying to edit the data (the path supported by the newest Entity Updates release of Central)

What version of Central are you trying this with? The latest release 2023.5 from a few weeks ago is the only release that supports Entity Updates via forms and submissions (path c) so I am imagining all of this in the context of that release.

In the morning (in California) I can work on getting you an improved example form, but I wanted to reply before then in case you have any other useful clues for me. Please let me know where I might be misunderstanding things about your scenario!

We really appreciate hearing how things are going with entities, especially the problems!

Finally, welcome to the forum! If you have a moment, please introduce yourself! I believe you are a colleague of @aurdipas?

Thanks,
Kathleen

3

Hi @ktuite
we have Central 2023.5.
But @kristen.jafflin is talking is more on updating the submission (not the entity itself).

thanks
Aurelio

4

Hi @ktuite,

Thank you for your reply. Please find the answer to your questions below.

"After playing with the example form, I think I understand what you are trying to do, but I wanted to double-check.

  1. If a data collector tries to enter data, check (based on an ID) if it already exists and stop them from re-entering data
  2. But also show the existing data associated with that ID and let them change it if it is wrong

So everything will happen through this one single form, but the form needs to be able to create an entity, check against existing entities, and also update an existing entity, yes?"

Not quite. The form stops people from re-entering data using the same ID unless they explicitly say they want to. If they want to correct data that is already entered, they are supposed to do so through a Project Manager account using the normal procedure (editing the appropriate submission from the submissions tab). It is during this editing process - editing the submission from the submissions tab - that I encounter the problem.

You asked:

"What is also probably adding confusion is the different things (submissions, entities) that can be edited and what does or doesn't have an effect on something else. When you 'open the entry for editing', are you:

a) editing the submission that first made the entity (editing a submission after it creates an entity doesn't affect the entity)
b) opening an entity edit pop-up in Central to edit the data of the entity (this is one way for an admin/manager in Central to fix entity data as they see it without a custom form)
c) opening the form to make a new submission, entering an ID to select an existing entry, and then trying to edit the data (the path supported by the newest Entity Updates release of Central)"

A. We are editing the submission that first made the entity. I want to edit the submission, not necessarily the entity itself. I use the submission data for analysis. The entity data exists to prevent double entry of the same submission or mistakes with identification numbers and to facilitate linking longitudinal data. It is not used for analysis.

You asked:

"What version of Central are you trying this with? The latest release 2023.5 from a few weeks ago is the only release that supports Entity Updates via forms and submissions (path c) so I am imagining all of this in the context of that release."

I am using the latest version of Central, but I am not trying to update entities via forms and submissions.

I hope this answers your questions, please let me know if I can clarify anything else.

Best regards,

Kristen

5

I think I understand now! Basically you need editing a submission to work differently than creating a new submission, especially how it relates to checking against existing entities.

You could check for a deprecatedID in your form to know if it's an edit or not, and combine that with your id_match and double calculations to determine if you should stop the user from editing the data or not.

(I'm not sure how the rest of the core team feels about this suggestion, I haven't seen any other examples of accessing deprecatedID in the XLSForm, but it is based on the submission updating system).

You would need to put /data/meta/deprecatedID (without quotes or ${}) in the calculation column of a calculate you wanted to store and use.

Here is an example form.
I made several of the calculate fields into notes just for visibility/debugging.
edit_prob_check_dep_id_20240112.xlsx (120.4 KB)

This example also saves the original submission instanceID as a property on the entity, and the compares the edit deprecatedID with the entity's source submission instance id so you can't edit a submission about a different entity. Unfortunately, this would only work for 1 edit because of how the instance id/deprecated ids kind of chain together with new submission versions.

You might be better off just checking if deprecatedID is empty or not and working that into your match logic.

1 Like
6

I now know this isn't what you were asking for, but your form had such rich logic for inspecting and changing the data of an existing entity that I wanted to show you what it would look like to actually make it update the entity.

edit_prob_example_update_entity_20240112.xlsx (120.1 KB)

This is similar to your original form, which can create entities, but if you enter an ID for something that already exists and you say you want to change the data, the form will be able to update the entity data (though a new submission rather than a submission edit).

The differences can mainly be seen on the entities tab and on one part of the survey tab where it gets the "name" (uuid) of the chosen entity.

1 Like
7

I’d like to add that this is a general class of problem that comes up any time you have a list of things, you want to filter them according to some properties (in this case by id to ensure uniqueness), and the form that’s being filled out changes those properties for the items (adds the id to the entity list in this case). Either during form filling, during edit on device or on the server, you’ll end up with the new property values which will change the value of existing computations.

As another example, sometimes people want to have a repeat with unique values input into each. They can use filters or constraints in a similar way than in this form. But they have to be careful when detecting a duplicate that they don’t include the value that was just entered by the user or it won’t be possible to enter any data at all. It’s the same with entities except that the problem manifests when doing a server edit, not when filling out the form initially because the change to the entity — in this case that it exists in the entity list — happens on the server.

@ktuite’s approach is a clever way of detecting that expressions are being evaluated in an editing context and changing behavior based on that.

Another approach would be guarantee that the calculation that filters the existing entity list by id to find an id match only runs if the entered id is changed by the user. With the current form definition, the calculation can be run at any time and when it's run on the server, it's run with the updated entity list that includes the entity that this submission created.

If you add ${id_match} to the trigger column for the confirm_id calculation, that calculation should not be run in the context of an edit as long as the user doesn't change the id_match field.

We know this is a common problem and that the existing solutions are awkward and not intuitive. It's something we will be working to improve over time.

2 Likes
8

Thank you, @LN and @ktuite for all your help! I've added ${id_match} to the trigger column as @LN suggested. That works well, and doesn't limit the number of times we can edit a submission. @ktuite, thank you for the example forms with a solution and for updating entities. I will look into integrating entity updates into the form, as it would be helpful to update the corresponding entity automatically to account for any edits made in the submission.

The form you provide looks great, but won't work in this case, as that option exists to account for an ID error in the original submission. E.g., if someone made a submission that should have had 212 as its ID, but they erroneously entered ID 221, when submission 221 is entered, it will be flagged as an existing entry. But I don't want to stop that submission from being made with the correct ID number, so they would check option 2 so that they could continue with the submission.

9

Ah, I understand the logic of your form now, thanks for clarifying!

E.g., if someone made a submission that should have had 212 as its ID, but they erroneously entered ID 221, when submission 221 is entered, it will be flagged as an existing entry. But I don't want to stop that submission from being made with the correct ID number, so they would check option 2 so that they could continue with the submission.

I'm sure there are ways to put this all together -- your form flow, entity updates, etc. -- if/when you get to that and have more questions, let us know!

Glad @LN's solution is working for you!

10

@ktuite thank you the example form.
This line was useful for me

calculate	entity_uuid  instance('edit_prob')/root/item[id=${id_match}]/name

I was doing something similar, downloaded my entity CSV and saw the column __id, so I was using that instead of name. For example:

calculate	entity_uuid  instance('edit_prob')/root/item[id=${id_match}]/__id

wasn't working.

In my form, I was using pulldata instead, I changed __id to name and it started to work.

Is it possible to get the __updates, __version or other of the metadata from the entity?

11

Hi @nmambre,

I'm so glad you were able to fix your problem. Thank you for pointing out this particular discrepancy!

The entity CSV that you can download directly and the entity CSV that gets used as a form attachment are slightly different. Mainly, the attachment CSV uses name for the unique entity ID column (called __id or Entity ID or uuid in different places) because name is what the select question types are expecting as a unique key. But this is not obvious and we need to improve our documentation around this and/or come up with a different solution.

To answer the other part of you question:

Is it possible to get the __updates, __version or other of the metadata from the entity?

Yes, __version is available in the attached entity CSV.

The other metadata (currently: __createdAt, __creatorId, __creatorName, __updates, __updatedAt) is not included.