What's coming in Central over the next few years

The Technical Steering Committee that guides the ODK suite has shared in our upcoming changes to Aggregate topic. Following these changes we are slowing development on Aggregate in order to concentrate on Central. We expect Central to become a viable replacement for Aggregate for most users by June 2020.

As we make progress towards that goal, we wanted to provide more visibility into our roadmap and timeline so you can offer feedback. We are particularly interested in the features Aggregate has that you need to switch to Central.

Please note that our goal for Central isn’t to build an exact copy of Aggregate. Rather, we want to make a new ODK server that works better for most users and is easier for our community to evolve.

As always, we’ll be able to move faster if we have your help. If you or someone you know can fund this work or help with the development, please email yanokwa@nafundi.com.

Features we support now

  • User accounts and management
  • Role-based user permissions
  • Projects to organize users, permissions, and forms
  • Form and submission upload and management
    • With support for form version updates
    • With drafts and testing on initial creation, and on version updates
    • With form and submission multimedia or data attachments
    • With a table preview of submission data
    • OData live data feed for analysis with tools like Excel and Power BI
  • Integrated checklist-based help system
  • Automatic, encrypted off-site data backups to Google Drive
  • Clean and modern REST API for integration and extensibility
  • High performance on low-cost hardware or cloud providers
  • ODK Briefcase-compatible data output
  • Encrypted forms (self-supplied or project managed keys)
  • Briefcase push/pull support
  • Upload XLSForms in addition to XForms
  • More fine-grained permissions for projects
  • Documentation for running on AWS
  • Download form XML and XLS
  • Preview forms in a web browser (powered by Enketo)
  • Fill out forms using Enketo

You can find out more about these features at https://docs.opendatakit.org/central-intro.

Short-term features

  • View all submission data
    • We currently let you see the first 10 fields in a submission. We’ll soon let you specify the fields (~30) you’d like to see in the user interface.
  • Documenting running on more cloud providers
    • We’ve provided documentation on Digital Ocean and AWS, but will add GCloud, and Azure.
  • Download form attachments
  • Filtering submissions by submission metadata (e.g., datetime received on server)

Long-term features

  • Sync the device to some centrally managed state
    • We’re aiming for a single button push that will sync forms, submissions, and settings.
  • Longitudinal data collection
  • Data editing/cleaning using Enketo (in partnership with @martijnr)
  • Delete all or some submissions, true deletion of forms
  • Bulk upload-download user email address and permissions
  • Improved install and upgrades to reduce dependence on command line
  • JSON-based webhook similar to Aggregate's JSON publisher
    • We have an OData feed
    • You can push/pull/export with Briefcase
    • De-prioritized until we have explicit demand for it

Features we’ll likely never add

  • Parsing submissions into database tables
    • Support for arbitrary schemas is a hard problem that introduces a lot of complexity (see Aggregate’s codebase).
  • Visualizing data as pie or bar chart or map
    • We believe this functionality should be pushed to external services through OData
  • Arbitrary database support (e.g., MySQL, SQLServer)
    • We believe PostgreSQL is the best database for our workload.
  • List of previously exported submissions
    • This shouldn’t be necessary since you can regenerate exports quickly
  • Toggling display of submission metadata
    • We will always show you all the relevant data.
  • Export to KML or some other format
    • We do not support KML, but GeoJSON is supported through Briefcase.
    • If there is a wildly popular export that users want, we’ll consider it, but we’d rather not add to our maintenance burden.
  • Publishing to Google Sheets or some other web service
    • The webhook should allow you to connect to other services that can then forward data on to Google Sheets.
    • If there is a wildly popular publisher that users want, we’ll consider it, but we’d rather not add to our maintenance burden.
  • Anonymous retrieval of images, audio, video for visualization
    • We expect downstream visualization tools to be able to securely auth against the server
  • Manually upload submission data
    • We believe this is a rare use-case and Briefcase supports it well.
  • Support of the Briefcase-Aggregate API
    • We believe the extensive RESTful API is better.
  • Running on Google App Engine
    • We value portability over the benefits of a platform as a service. You will be able to use Google Cloud, just not Google App Engine.
  • A non-Docker install
    • We use standard components, so it’s possible to set up Central outside of Docker. We don’t recommend it because we don’t have the resources to support non-Docker installs.

Some considerations to be aware of

If you use App Engine, there’s currently no way to redirect traffic from an appspot domain to another domain. You'll need a plan for changing the server URLs on your devices. If you use your own domain (e.g., with the Aggregate v2.0 or later), that domain can be used with Central.

Our expectation is that most users will keep using Aggregate for ongoing campaigns and will start new campaigns with Central. That is they won’t migrate their data. If migration is needed, Briefcase will soon be able to migrate blank forms and submissions from Aggregate to Central. Publishers, users, filters, etc will not be moved.

Our goal is to make Central fast, secure, and easy to use. Using cloud hosting is the best way for our small team to reach that goal. Unfortunately, cloud hosting these days isn’t free, so you should expect to pay $5-10/month. If $5/month isn’t affordable, then you can host on your hardware and we’ll provide some documentation, but we’ll be limited in the support we can provide. Contributions here are very welcome.

This timeline is not set in stone! We’ll continue to update this post and the timelines as we get new information, so please watch this topic to get updates.

I have a question or comment

Put it below :point_down:


Should these be captured in Server Logs though? So for example I as an admin can know that @chrissyhroberts as a project manager downloaded the data at a certain date-time?

1 Like

@dr_michaelmarks presently this is not planned.

but i know for hipaa / similar reasons it can be a requirement so i'm not opposed to adding it someday.

1 Like

Just a heads up that I've updated the initial post with our progress. The recent changes are:

  1. Upload XLSForm forms has shipped.
  2. Docs for AWS installs are available.
  3. Form versioning and drafts are in progress and will ship in March.
  4. Enketo form preview support will likely ship in July.