Filter Data in ODK Central

[Matthew_White]

@LN just wrote up a detailed description of our thinking around what Central can offer in terms of data analysis. I wanted to highlight a couple of paragraphs that I think are especially relevant to this topic:

@LN

One of the early decisions we made was to store incoming form submissions as XML blobs rather than splitting them into database tables as Aggregate does. This is a decision that we did not make lightly. It has helped our small team make quicker progress and has ensured speed and stability as submissions come in. We learned from Aggregate and other systems that splitting records is a big source of code complexity, bugs, and performance bottlenecks.

The tradeoff is that this limits the analysis that can be done directly on Central -- the data is not organized for any kind of fast operations across the dataset. Additional implications are that directly connecting to the Central database for analysis is not practical and that we don't provide a performant API for open-ended data querying. … [W]hat we learned from Aggregate is that most people need to rely on external tools for analysis anyway.

I also wanted to link to What's coming in Central over the next few years, which discusses future directions for Central.

@dmenne

Example 1: Get all records from patient 1234, and create a time series of the blood cell count.

@dmenne, I'm realizing that your example may relate to ongoing discussions around entity-based data collection. It seems possible that entity-based data collection would simplify your workflow. Related to that is this section from @LN's post:

@LN

As we explore more managed workflows for entity-based data collection, it is possible that we will introduce an entity concept that is more richly queryable. However, this is unlikely to become our immediate focus because it is an entirely new area of work. Additionally, what may look like simple functionality can be complex or computationally expensive to do on large datasets. We’d like to first strengthen what can be done with web-based forms (e.g. submission edits), improve and enrich the user and permissions model, and make sure the features we already have are polished.