Coming soon: powerful new web forms for ODK!

The ODK team is building a new web forms system to better serve your needs! As part of that process, we'll be transitioning out of maintaining Enketo web forms by December 2024.

Why are we making this decision now?
The ODK team became the primary maintainers of Enketo in August 2021. We made that choice because Enketo is a key component of the ODK ecosystem for submission editing and web-based surveying.

Our goals in maintaining Enketo were to add new features, modernize the user experience, and fix long-standing bugs, all without disrupting existing users. While we’ve made good progress, the pace has been slower and more difficult than we anticipated.

A few months ago, we started a technical discovery process to see if we could more quickly address your needs by building an entirely new web forms system we call ODK Web Forms. We were able to quickly build features in this new engine, and so after much consideration, we have decided to transition from using Enketo to ODK Web Forms.

Our goals for ODK Web Forms
We believe that web forms are valuable because they can be used on any device (including iOS) and allow us to serve different types of data collection campaigns (e.g., online market research, self-report studies) that many of you have been asking for.

This is why we are investing in a new system that allows greater speed, reliability, and security as we build new features. ODK Web Forms will also allow us to both build new user experiences in Central (e.g., form testing) while enabling us to create a data collection experience which is more aligned with Collect.

What have we built so far?
We have built a strong foundation that can quickly render large and complex forms with many of the common form features. If you’re a developer, you can go to our GitHub and try it yourself. The read me has more information about project status and reasoning for some of the decisions we’ve made. If you aren’t a developer, we’ll soon share ways you can try Web Forms and offer your feedback.

What’s next?
We want to be very thoughtful about how this change is rolled out so it doesn’t disrupt or slow your important work. To do that, we’ll need your feedback as we build this exciting new future for ODK. Some of the areas we will be working on next are:

  • :technologist: Building low-risk ways for you to try Web Forms and provide feedback
  • :rocket: Supporting more form features and question types
  • :ring_buoy: Ensuring a seamless transition for you and your data collectors

What will happen to Enketo?
For some organizations, Enketo remains a great web forms solution and they may want to continue using it as long as there are security updates, etc. We’ll continue ongoing maintenance on Enketo as needed until the end of 2024. We are confident we can find another organization that will continue maintenance after and you can follow the search for a maintainer here.

We are open to feedback and questions in this thread. If you have any private concerns, email me at yanokwa@getodk.org.

12 Likes

Very exciting news! The collapsing repeat nodes is a nice touch too.

Keen to see how images / annotations and geo / map tiles will be handled differently to Enketo

4 Likes

We are working on some of these features as we speak and would love to get feedback. We will share more soon!

3 Likes

A very exciting endavour, I will be more than happy to see how the new odk webform incorporates key features like theme-grid style- which makes it efficient to collect data in a spreadsheet like format. The very feature missing in odk collect. Also, the offline capabilities will be of interest to have.

1 Like

One of the things we intend to do is to allow grid presentation to apply on individual groups as well as across a full form. If there are specific requirements you have for this kind of presentation, it would be great to hear them.

It will likely be some time before we get there but we have some ideas for how we will approach offline capabilities. @Aly_Blenkin will be leading some research efforts to make sure we have a thoughtful design!

4 Likes

The offline features of Enketo are an important feature for many users in remote (humanitarian) areas. So, support for Enketo would be needed until similar and well tested offline options are available for your new Webform product.

Could you provide some information, please,

  • how you imagine/plan the (automatic) transformation for existing Enketo applications?
  • if the new Webtool will be free of charge within Central?
  • if the new Webtool will be available in the (partly free of charge) systems of KoboToolbox and ONA?

We were surprised to read the announcement about the end of your Enketo support this year, as we could not remember having read about this in any roadmap or forum discussion or TAB meeting before. Maybe, we missed something? And it was strongly caused now by the "entities" development?

We are also afraid, that (after "Entities") this will block a lot of your development resources all over this year. We were glad before when you took over the Enketo support, but now, there might finally come out more separated systems, tools, developer teams, fora, documentations around XLSForm again.

1 Like

No, it's not related to Entities. We have some ideas on how we'd like to enable offline web-based access to Entities across forms once we get there but that is a while away.

We have invested a lot in Enketo and made some significant improvements but have not managed to reach the pace and confidence that we wanted. Some of that has to do with Enketo's age -- patterns and libraries have evolved -- and some is simply that the code base's structure is not a good fit for our team and our needs.

You and other community members have requested fixes for deep bugs in web forms, improved performance, and greater consistency with Collect. We tried different approaches with Enketo and then decided to explore a fresh start. Shifting to a new web form system is a big task with significant risk but we think it will be worth it to better serve those with complex web form needs.

We generally aim for high transparency into our processes. In this context, we really weren't sure what we were going to do and took some time to explore options internally and gain confidence before opening things up.

We did not solicit public feedback on the high-level decision to transition from Enketo to a new web form system because the decision is driven by our current development capacity. We did discuss with Martijn, Enketo's creator, and other major stakeholders as soon as we thought this might be a possibility. Most of our discussions around design considerations for the new system are now public on Github and summarized in the readme.

We will be sure to solicit feedback as we plan this. Our preliminary thinking is that we will give users the ability to opt in to new web forms at the form level. We also intend to build in form feature detection so that we can warn or error when a form attempts to use features that aren't yet supported.

The ODK team has always produced 100% open-source software that we try hard to make easy for others to use and adapt. Web Forms is no different and is being produced under our usual Apache 2.0 license. We don’t require contributions (financial or otherwise) for people to use ODK. This is not easy to sustain and we are grateful to the wonderful customers and supporting organizations whose funding ensures we can keep making ODK.

These are legitimate concerns! We believe that this approach will be less resource-intensive than continuing what we had been doing with Enketo. We also believe that greater alignment with Collect will be beneficial in the longer term. There will certainly be a challenging transition period and we will do our best to smooth it out as much as possible with the help of community input and support. We've worked extremely hard to make ODK a sustainable and vibrant project and intend to keep doing our best to continue improving it.

8 Likes

That is a surprising news but I understand you points; I do have one question:
will those form be based on XForms or another type of definition ?

best regards

Yes, New Web Forms will be based on ODK XForms.

For the benefit of the larger audience, I am attaching the Form that we used for creating demo video:

ses2018.xlsx (34.6 KB)
ses2018.xml (29.5 KB)

4 Likes

The expand a little bit on @Sadiq_Khoja's response -- there are many things we would likely do differently if we were designing a form spec from scratch but we strongly believe that a consistent, shared, well-documented form spec (warts and all!) has been a key to the longevity and vibrancy of the ODK ecosystem.

We did explore operating directly on XLSForms which has become the user-facing form standard but it really is specified as a form editing format and all of the form behavior defers to the underlying ODK XForms spec. Using the underlying ODK XForms spec means we can easily leverage existing fixtures/tests from Enketo and JavaRosa to align behavior. If you're curious, you can see that @eyelidlessness has ported over almost all JavaRosa tests at https://github.com/getodk/web-forms/pull/110. This gives us a really clear understanding of what does and doesn't work so far.

One of the many wonderful things about Enketo is that it is general-purpose and can be used in a broad range of systems. @Tino_Kreutzer asked a question about how general-purpose this library will be compared to Enketo. ODK Web Forms is best compared to Enketo Core, a frontend library that can be embedded in host web applications. Similarly to Enketo Core, we intend to make it possible to embed ODK Web Forms in any web application, likely by publishing a custom web component.

ODK Web Forms will not take on any of the responsibilities that the Enketo Express service has -- things like URL management, OpenRosa API implementation, offline capabilities. Some of these we'll provide reference implementations for. Others we think won't be needed by most embedding systems or will be straightforward to implement within most applications.

ODK Web Forms will publish its engine entirely separately from its frontend. This is in contrast to Enketo Core which does all its work on and within HTML. The first step to use Enketo Core is to transform an ODK XForm to an HTML representation. The ODK Web Forms engine will operate more directly on form definitions and will expose a reactive model of the form. This structure will make it possible to use this engine to, for example, drive a mobile application like Collect. It also means that authors of a host web application could choose to implement their own frontend using tools and styling of their choice.

There is some more information on these themes available in the readme. Let us know if you have other questions!

2 Likes