Using Entities on larger projects

It's unfortunately really hard to make definitive statements about performance because there are so many factors involved.

We give 50k as an estimate of when performance degradation begins based on two major factors: device RAM and transfer connection. These both play a big role because currently the entire Entity List must be stored in memory and also be transferred with each update.

Both are also heavily affected by the width of the data: how many properties there are and how long their values are on average. Speed of transfer will also be affected by how repetitive the data is. That is, if you have many many properties that each have values 'yes' or 'no', that will get compressed down to a much smaller transfer size than a list with few properties that each have unique values.

RAM usage will change significantly with Collect v2024.3.0 which is currently in beta as we work through some remaining issues. This version will store Entity Lists in a database and significantly reduce memory needs for larger Entity Lists. The whole list will still need to be transferred and processed, however.

All this to say, you may be able to have significantly more than 50k rows depending on the shape of your data. I would encourage you to generate sample data of the shape expected in a particular project, upload it as an Entity List and try empirical checks with the devices you will use. Or if you tell us more about the shape you expect and the devices you will use, we may be able to give you a sense of what's possible.

Yes, absolutely.

Yes, we haven't optimized displaying multiple points with the assumption that generally data can be filtered.

That sounds great. You could have a cluster property and a choice filter for that.

Yes, absolutely.

It's certainly not as smooth as we eventually want it to be! As you've described, with some creativity it's possible to make it work, but you need to decide whether the manual interventions (splitting up the data, automating Entity removal, etc) are in reach and worth it for you.

1 Like