Entities and Datasets in ODK Central

I really like the new feature where you can link new questionnaires to an original dataset. I have a few questions: how many datasets can be made? And also - how easy is it to find your specific ID when there are a few hundred or a few thousand?

For example, if I have a longitudinal cohort, can I have multiple different datasets/questionnaires that are are linked to the registration?

Thank you!

1 Like

Thanks for trying out the entities preview and for sharing your feedback!

There's no hard limit. There should be no performance implications to having many datasets. Currently, there are performance implications to having many entities but that's mostly on the client side (Collect or Enketo) and in data transfer. Those bottlenecks will prevent you from realistically handling more than a few tens of thousands of entities in a single dataset.

There are a few ways you can select a specific entity in a form.

The most scalable approach is to have a unique identifier that can be scanned or entered to filter out the exact desired entity. You can see an example of how to do that in the docs. To use the new entity support, you would modify the registration form to create entities with at least the id field turned into a property. Then your follow-up form would attach the server-managed dataset and otherwise look just like the example form. Please let us know if you'd like a worked example.

If it's not practical or possible to have guaranteed unique identifiers for each entity, you have a couple of options you can use to make entity selection easier. These can be used separately or combined. First, you can use questions in your follow-up form to filter down your entity list. For example, your registration form could capture the town and the color of the registered entities. Then your follow up form can ask for the town and the color of the entity to look up and use a choice filter to filter based on the town and color properties. You can also use the autocomplete appearance on a select to allow data collectors to progressively filter down the list of choices.

You can also do things such as ask "does the participant have their barcode today?" and if so, go through a barcode scanning flow. If not, you can provide other lookup options. Feel free to describe your scenario in greater detail and we can work on a sample form together.

Currently a registration form can only populate one dataset. Eventually we will expand that. Any form, including a registration form, can have any number of datasets attached to it. You can have a form that both creates entities in a dataset and consumes entities from that same dataset. Note, however, that currently there is no on-device dataset update. Let's say that formA registers entities in datasetA and also attaches datasetA. You fill out formA to register entityA. You won't see entityA in formA until you've submitted your filled out formA to Central and your client updates its attached dataset.

4 Likes

Thank you Jordyn for your message, it helped me get the news that entities are finally available in central now. Waao! what a great news.
I will be trying it soon.

4 Likes

Thank you for your response! Are there any plans to extend this so you can have different questionnaires/datasets linked to the registration?

Wonderful, we look forward to hearing what you think about what is there so far and how you'd like to use it! The documentation is a good place to start.

Can you please describe what you want to do in a bit more detail? I'll try to give some information that will answer your question but I would find it helpful to learn more about what you have in mind.

You can currently link as many datasets as you want to any kind of form including a registration form.

If a form registers entities in a specific dataset, it can also attach that same dataset and access its entities. This can be useful to do things like identify whether a given entity was previously registered. Note that currently entities are not created offline so entities you have registered won't be available until they've been submitted, the submissions have been approved, and the device has updated its dataset.

Currently a form submission can only create at most one entity. In the future it will be possible to register multiple entities at once either through a repeat (multiple entities in the same dataset) or not (multiple entities possibly in different datasets).

You may also find some of the discussion and ideas at Entities - get the most recent record from the database useful.

Thank you @LN for your detailed reply.
Yes, I intend to use it for a project dealing with farmer training.
We register learning groups which will create dataset.
Then farmers will be registered and learning group dataset will be called to attached LG information with farmers. The farmer registration will create another dataset of farmers.
Later on those farmers will be interacted on intervals to see their level of learning and agriculture practice adoption. so in the followup forms, farmer dataset will be called.
I am working on this flow and excited to use the new feature.

I read that the editing of entities in a dataset will come in the future versions. I will be waiting for that as well.

But even the current modality looks really really useful for our usecase.
Thank you for your great work and hats off to the whole team.

3 Likes

This is a great add on feature I've waiting to see added on to odk ( ..and I believe many others have been waiting for ...), more or less like a case management feature in odk-x/ commcare case mngt. This will really make it quite easy to manage and do follow up for cases in longitudinal studies after registration of cases using unique IDs. Heading straight to documentation to see the implementation and try it out.
Big up to the team for adding such a great and useful feature. :ok_hand::100::partying_face:

1 Like

Hi @LN
I hope you are doing very well.
I started using the feature and I need your support for the following answers:

  1. we create a registration form to create a dataset AA. If the submission is edited later and approved; will that update the dataset AA automatically or not?

  2. is there a way to edit the dataset entries? any future plan?

Thanks a lot for your excellent work on this feature, hats off to the whole team.

I submitted a form which created an entity.
Later I edited the entity main field which is supposed to work as entity label. I approve it. Its saved fine but later the screen says Problem creating Entity. Please see the image below:
Any thoughts???

1 Like

Hi @Syed_Muhammad_Qadeer,

Thank you for giving this a try and sharing your feedback! Please note all things Datasets and Entities are still a preview, so there are many things to fix, improve, and add.

We have not yet implemented the ability to edit/update/remove an entity (dataset entry) but it is one of the things we are working on next. It's a top priority to not have users be stuck with incorrect entity data in their datasets.

What I believe is happening in the specific "Problem creating Entity" scenario you posted above is that Central is trying to re-create a new entity from the edited submission, but failing because the UUID of the entity already exists (from the first time you approved the submission and created an entity). This will always fail, so basically Central should not even try to make an entity if an entity already exists for a submission. We will change this!

Also, it seems like a bug that the error message is not shown. I can recreate this error, and when I look at the network traffic in my browse console coming through to that submission page, and read the 'audits' JSON, I can see a the a problem with code 409.3 and message "A resource already exists with uuid value(s) of [some uuid]." We can fix this to make sure the message does come through on the website.

My main recommendation is that, for now, realize that you can edit a submission before approving it, and those changes will be reflected in the entity, but editing a submission after an entity has been created will have no effect on that entity. We will soon introduce the ability to edit entities!

3 Likes

Thank you. That is a super helpful explanation.

1 Like