Coming soon: powerful new web forms for ODK!

Ideas
1

The ODK team is building a new web forms system to better serve your needs! As part of that process, we'll be transitioning out of maintaining Enketo web forms by December 2024.

Why are we making this decision now?
The ODK team became the primary maintainers of Enketo in August 2021. We made that choice because Enketo is a key component of the ODK ecosystem for submission editing and web-based surveying.

Our goals in maintaining Enketo were to add new features, modernize the user experience, and fix long-standing bugs, all without disrupting existing users. While we’ve made good progress, the pace has been slower and more difficult than we anticipated.

A few months ago, we started a technical discovery process to see if we could more quickly address your needs by building an entirely new web forms system we call ODK Web Forms. We were able to quickly build features in this new engine, and so after much consideration, we have decided to transition from using Enketo to ODK Web Forms.

Our goals for ODK Web Forms
We believe that web forms are valuable because they can be used on any device (including iOS) and allow us to serve different types of data collection campaigns (e.g., online market research, self-report studies) that many of you have been asking for.

This is why we are investing in a new system that allows greater speed, reliability, and security as we build new features. ODK Web Forms will also allow us to both build new user experiences in Central (e.g., form testing) while enabling us to create a data collection experience which is more aligned with Collect.

What have we built so far?
We have built a strong foundation that can quickly render large and complex forms with many of the common form features. If you’re a developer, you can go to our GitHub and try it yourself. The read me has more information about project status and reasoning for some of the decisions we’ve made. If you aren’t a developer, we’ll soon share ways you can try Web Forms and offer your feedback.

What’s next?
We want to be very thoughtful about how this change is rolled out so it doesn’t disrupt or slow your important work. To do that, we’ll need your feedback as we build this exciting new future for ODK. Some of the areas we will be working on next are:

  • :technologist: Building low-risk ways for you to try Web Forms and provide feedback
  • :rocket: Supporting more form features and question types
  • :ring_buoy: Ensuring a seamless transition for you and your data collectors

What will happen to Enketo?
For some organizations, Enketo remains a great web forms solution and they may want to continue using it as long as there are security updates, etc. We’ll continue ongoing maintenance on Enketo as needed until the end of 2024. We are confident we can find another organization that will continue maintenance after and you can follow the search for a maintainer here.

We are open to feedback and questions in this thread. If you have any private concerns, email me at yanokwa@getodk.org.

19 Likes
3

Very exciting news! The collapsing repeat nodes is a nice touch too.

Keen to see how images / annotations and geo / map tiles will be handled differently to Enketo

4 Likes
4

We are working on some of these features as we speak and would love to get feedback. We will share more soon!

4 Likes
5

A very exciting endavour, I will be more than happy to see how the new odk webform incorporates key features like theme-grid style- which makes it efficient to collect data in a spreadsheet like format. The very feature missing in odk collect. Also, the offline capabilities will be of interest to have.

2 Likes
6

One of the things we intend to do is to allow grid presentation to apply on individual groups as well as across a full form. If there are specific requirements you have for this kind of presentation, it would be great to hear them.

It will likely be some time before we get there but we have some ideas for how we will approach offline capabilities. @Aly_Blenkin will be leading some research efforts to make sure we have a thoughtful design!

4 Likes
7

The offline features of Enketo are an important feature for many users in remote (humanitarian) areas. So, support for Enketo would be needed until similar and well tested offline options are available for your new Webform product.

Could you provide some information, please,

  • how you imagine/plan the (automatic) transformation for existing Enketo applications?
  • if the new Webtool will be free of charge within Central?
  • if the new Webtool will be available in the (partly free of charge) systems of KoboToolbox and ONA?

We were surprised to read the announcement about the end of your Enketo support this year, as we could not remember having read about this in any roadmap or forum discussion or TAB meeting before. Maybe, we missed something? And it was strongly caused now by the "entities" development?

We are also afraid, that (after "Entities") this will block a lot of your development resources all over this year. We were glad before when you took over the Enketo support, but now, there might finally come out more separated systems, tools, developer teams, fora, documentations around XLSForm again.

1 Like
8

No, it's not related to Entities. We have some ideas on how we'd like to enable offline web-based access to Entities across forms once we get there but that is a while away.

We have invested a lot in Enketo and made some significant improvements but have not managed to reach the pace and confidence that we wanted. Some of that has to do with Enketo's age -- patterns and libraries have evolved -- and some is simply that the code base's structure is not a good fit for our team and our needs.

You and other community members have requested fixes for deep bugs in web forms, improved performance, and greater consistency with Collect. We tried different approaches with Enketo and then decided to explore a fresh start. Shifting to a new web form system is a big task with significant risk but we think it will be worth it to better serve those with complex web form needs.

We generally aim for high transparency into our processes. In this context, we really weren't sure what we were going to do and took some time to explore options internally and gain confidence before opening things up.

We did not solicit public feedback on the high-level decision to transition from Enketo to a new web form system because the decision is driven by our current development capacity. We did discuss with Martijn, Enketo's creator, and other major stakeholders as soon as we thought this might be a possibility. Most of our discussions around design considerations for the new system are now public on Github and summarized in the readme.

We will be sure to solicit feedback as we plan this. Our preliminary thinking is that we will give users the ability to opt in to new web forms at the form level. We also intend to build in form feature detection so that we can warn or error when a form attempts to use features that aren't yet supported.

The ODK team has always produced 100% open-source software that we try hard to make easy for others to use and adapt. Web Forms is no different and is being produced under our usual Apache 2.0 license. We don’t require contributions (financial or otherwise) for people to use ODK. This is not easy to sustain and we are grateful to the wonderful customers and supporting organizations whose funding ensures we can keep making ODK.

These are legitimate concerns! We believe that this approach will be less resource-intensive than continuing what we had been doing with Enketo. We also believe that greater alignment with Collect will be beneficial in the longer term. There will certainly be a challenging transition period and we will do our best to smooth it out as much as possible with the help of community input and support. We've worked extremely hard to make ODK a sustainable and vibrant project and intend to keep doing our best to continue improving it.

9 Likes
9

That is a surprising news but I understand you points; I do have one question:
will those form be based on XForms or another type of definition ?

best regards

10

Yes, New Web Forms will be based on ODK XForms.

For the benefit of the larger audience, I am attaching the Form that we used for creating demo video:

ses2018.xlsx (34.6 KB)
ses2018.xml (29.5 KB)

5 Likes
11

The expand a little bit on @Sadiq_Khoja's response -- there are many things we would likely do differently if we were designing a form spec from scratch but we strongly believe that a consistent, shared, well-documented form spec (warts and all!) has been a key to the longevity and vibrancy of the ODK ecosystem.

We did explore operating directly on XLSForms which has become the user-facing form standard but it really is specified as a form editing format and all of the form behavior defers to the underlying ODK XForms spec. Using the underlying ODK XForms spec means we can easily leverage existing fixtures/tests from Enketo and JavaRosa to align behavior. If you're curious, you can see that @eyelidlessness has ported over almost all JavaRosa tests at https://github.com/getodk/web-forms/pull/110. This gives us a really clear understanding of what does and doesn't work so far.

One of the many wonderful things about Enketo is that it is general-purpose and can be used in a broad range of systems. @Tino_Kreutzer asked a question about how general-purpose this library will be compared to Enketo. ODK Web Forms is best compared to Enketo Core, a frontend library that can be embedded in host web applications. Similarly to Enketo Core, we intend to make it possible to embed ODK Web Forms in any web application, likely by publishing a custom web component.

ODK Web Forms will not take on any of the responsibilities that the Enketo Express service has -- things like URL management, OpenRosa API implementation, offline capabilities. Some of these we'll provide reference implementations for. Others we think won't be needed by most embedding systems or will be straightforward to implement within most applications.

ODK Web Forms will publish its engine entirely separately from its frontend. This is in contrast to Enketo Core which does all its work on and within HTML. The first step to use Enketo Core is to transform an ODK XForm to an HTML representation. The ODK Web Forms engine will operate more directly on form definitions and will expose a reactive model of the form. This structure will make it possible to use this engine to, for example, drive a mobile application like Collect. It also means that authors of a host web application could choose to implement their own frontend using tools and styling of their choice.

There is some more information on these themes available in the readme. Let us know if you have other questions!

3 Likes
12

Are there any plans to rework the map widget? Because its usability in my opinion is pretty awkward. I would be very happy to see a map widget that does not require to access the map on a "sub-screen-level", mark a place, then go back to the survey (the back-arrow is definitly not intuitive) and then not having a feedback if the point was saved because the map is not visible anymore (i usually trigger a note here, but the visual feedback where the point has been set is still missing).

Also, when panning the map in a multiple pages setup pretty often it happens that you jump back to the previous page.

These two points are also constantly pointed out by users in evalutions we did and i totally share this view.

So if a had two whishes free for the upcoming webforms it will be:

  1. a map that can be used directly from the survey screen (maybe with an option to toggle the map to full screen on demand by users)
  2. fixing the problem of unintended swiping back when panning the map.

Really looking forward to try the new webforms. And thanks for carrying this on!

2 Likes
13

Hi @Daniel_S, I'm glad you asked because we are planning on improving the map user experience for Collect and web forms :slight_smile:

Would you be open to doing a user feedback call with me in the next few weeks? These pain points are insightful and I would love to hear more about your workflow to see what else we can improve.

User feedback for geo functionality :speech_balloon: If anyone else has similar pain points or ideas to improve maps for Collect or the new web forms, we would love to hear it!

2 Likes
14

The current map implementation in Enketo is a significant hurdle for me being able to use it to create it edit records so it's really encouraging to hear this is a focus. I'm also happy to chat (in a week or two) if it helps.

2 Likes
15

It would be great to chat. I will message you to find a time.

16

I'd be happy to jump on a call about the map widget as well. Some of my users struggle to use it in it's existing style

1 Like
17

I have not been able to use Enketo for my use case becauseof the awkward, tiny map. Unfortunately that meant choosing between grid-themes and map widgets. Maps won out, but boy are our forms convoluted. Really excited to hear both of these will exist in one place now. Please reach out if you would like a user feedback call. I have tons of ideas :slight_smile:

2 Likes
18

Welcome @rachelthoms –– we would love to hear your feedback and ideas. I'll reach out to find a time.

When you get a chance, please introduce yourself on this forum thread!

19

Dear ODK Developer Team,
I Was very happy to test this very innovative new features called Web forms. I never program a questionnaire in ODK as for today. As a CSProuser, I always need flexibility and a level of control over any questionnaire application, that ODK don't offer. This will change with the new Web-forms. So, I would like you to improve by:
a) Giving us a way for input to be sized accordingly to item size. As an example, please take a look on CSPro questionnaire viewer:https://www.csprousers.org/help/CSPro/view_questionnaire.html, https://www.youtube.com/watch?v=70F1oCjGA8Y, https://www.youtube.com/watch?v=AbuntOPmays, https://www.youtube.com/watch?v=qC18dKmJrlM

b) Allowing to post data as JSON, not only as XML for ODK related server. If you give us JSON, we'll be able to save them in relational schema on device using this new embedded PostgreSQL version : https://github.com/electric-sql/pglite. We'll also be able to sync/replicate on centralized Postgre servers. Finally it will be possible to have ODK Collected data in a relational schema.

c) Giving us a more convenient way to embed an ODK Web Forms to a website.
Please take a look on on this testing version: https://getodk.org/web-forms-preview/app/index.html#/form?url=https://staging.xlsform.getodk.org/downloads/d16ba037bdaa427f9af98cc46d1de0b4ol1z57qb/Questionnaire_Canaan_ODK_16Fev2018.xml

Thanks in advance,

20

Thanks for commenting, we're glad to hear that you're liking what you see with Web Forms so far!

Can you please say a little bit more about what you mean here. Are you referring to text fields and their width? The ability to lay out questions in a grid?

It would be helpful to better understand what your end goals are. One of the design decisions we've made with ODK and one that is a strength for many projects is that form definitions can evolve as needed. That leads to a dynamic schema, possibly with nesting (repeats).

Some users build read-only replicas of their data in Central in various ways, including postgres functions. You need to be really careful to understand the limitations of those sync tools and how they apply to your context (e.g. do you make submission updates, do you use submission rejection, etc).

If you need a JSON representation, we recommend requesting that from Central: https://docs.getodk.org/central-api-odata-endpoints/#data-document

1 Like