The restriction on 50,000 entities is hindering certain projects aimed at updating entities within large populations. We are using this for patient follow-up updates and vaccination tracking. Please give a solution or increase this limit.
We are actively working on addressing these performance limitations and expect significant improvements by late 2024. In the mean time, one possible workaround is to use pulldata and search instead of instance and select_one_from_file. These methods are less flexible but they will perform better.
-- https://docs.getodk.org/entities-intro/#can-i-have-millions-of-entities
1 Like
Just out of interest: what number of entities are you looking to support? Hundreds of thousands? Millions? More?
Hi, We are working with big population so entities also in millions and more. Any update regarding increasing the limit of 50,000. Or any other way to manage.
The next release of Collect will add support for larger sets of entities (when used with the latest Central v2024.2.0). We'd hope that hundreds of thousands would be more feasible, but it'll depend on the forms using them (not all predicates will work with larger datasets yet for instance). I'm guessing that millions may still not be possible.
And just to be clear, the limit is not something we're enforcing. The larger datasets (including entity lists) are the more resources Collect will need to consume and the slower filter expressions will be. We're actively working on making this better to support larger datasets and make filtering them faster (it's part of our roadmap), but it's not something we can flip a switch to activate sadly.
I'd encourage taking a look a the latest Collect beta to get a sense of the changes coming.
2 Likes