Integration of Enketo into ODK Central

I'd like to propose and gauge interest in the following 3 types of additional functionality to ODK Central using Enketo's API.

Viewing/pre-viewing/pdf-ing forms

Various ways to view empty forms as well as records (loaded into a form). For example:

  1. A form preview to test form appearance and logic, as used on https://opendatakit.org/xlsform/ (using this API call)
  2. A PDF of an empty form (using this API call)
  3. A PDF of a record loaded into a form (using this API call)
  4. Viewing a record in readonly form (using this API call)

Editing already submitted data

This is the feature that KoBoToolbox, Ona (and even the old Formhub) are using to edit submitted data in a browser (using this API call). There have been some expressions of interest in this feature (for Aggregate) in this forum. Technically, this requires the ODK Central backend to become able to deal with replacing earlier submissions (using an instanceID -> deprecatedID swap). This could be done by either maintaining a chain of edits or by removing an older version of a record once an edit has been submitted.

Collecting data in a browser

This would expose additional use cases/features for collecting data with ODK Central such as:

  1. online website-embedded surveys (using this API call)
  2. email single-use surveys that have some protection against multiple submissions by the same person (using this API call)
  3. online surveys where default values for questions can be dynamically populated (per user if desired)
  4. desktop/laptop data collection/entry
  5. iOS data collection/entry
  6. displaying questions in a flexible grid using the Grid Theme

As well as providing an alternative to ODK Collect for

  1. Offline-capable, multi-submission surveys done by enumerators (using this API call)

If some of these seem interesting to you, please let us know! The feedback can be used for fund-raising and planning.

4 Likes

Hi @martijnr thanks for this. Being able to test xlsxforms appearance and logic would be great for a lot of reasons; most of all to new form developers. It would have saved me lots of time when I first started learning what different question types, constraints, and relevant statements did.

I also think having an easier mode to collect surveys in the browser would be great. At my organization we use Google Forms or Survey Monkey for this, and frankly, their logic and constraints have a lot to be desired. I'm always left frustrated with how I can add quality control mechanisms or skip pasterns, even in relatively
basic surveys.

1 Like

@martijnr Would these features rely on Enketo's hosted service? If so, could we bundle Enketo with Central?

2 Likes

Good question. That dependency would have been clever! :slight_smile: But no, it wouldn't rely on a hosted service.

It is best to keep them in separate boxes like Aggregate/Enketo, KoBo/Enketo, Ona/Enketo, so we can use the same loosely-coupled Enketo Express app for all. The link between them is established with 2 config items (API URL and API key).

I haven't yet looked into ODK Central deployment, but I believe it's using docker (which is something I have not used myself). So it could be 'bundled' as an (optional) docker container on the same server or placed on a separate server (KoBo is using docker to deploy Enketo on separate servers). For any serious traffic it would need its own server. The demands on CPU and memory can become fairly high (though odk.enke.to is still running reasonably well on a $5/month DigitalOcean box to my surprise).

Update: It is maybe good to give an idea of the (manual) installation procedure of a production server: https://blog.enketo.org/install-enketo-production-ubuntu/

2 Likes

FYI I use Enketo extensively for this purpose (from KoboToolbox), basically to test my new forms while I'm writing them. it saves me a tremendous amount of time over the alternative of having to export the XForm, import it into my sever, download it onto my device, fire it up, ...

I'd also love to be able to reuse it to generate a PDF version of a completed form, but unfortuantely this is something I have to do offline on my the device, so I'm having to roll-my-own [FYI I actually convert my completed XForm to HTML, and let the device's web viewer do the PDF conversion]. So an open-source, Enketo-based XForm2PDF library would be awesome... hint, hint :slight_smile:

4 Likes

We have lots and lots of users at LSHTM so here are comments from our perspective:

Viewing/pre-viewing/pdf-ing forms
All four of these would be extremely helpful.
Specific use cases

  1. Form preview very helpful for development

  2. PDF versions are very useful when training staff on questions and also for auto-generating a document we can submit to ethics committees who often want to 'see' the data collection form even when it doesnt exist as a paper document (and who couldn't handle looking at an excel)

  3. Again PDF records in read-only form would be hugely helpful (can be required in some kinds of study for governance reasons)

Editing already submitted data
This would be valuable.
Especially if it did so in a way that maintained an audit trail (i.e so you could see which user edited what field at what time and what the previous values were - That would be co-aligned with my request for anaudit trail within collect

Collecting data in a browser
Again this would be very desirable

@chrissyhroberts thoughts wlecomed

1 Like

Yes very useful for training especially using the guidance_hint functionality.

3 Likes

Did this progress since these posts?

We are interested in using Enketo for a form preview to test form appearance and logic, for a PDF of an empty form, and for collecting data in a browser (specifically desktop/laptop data collection/entry, and as an alternative to ODK Collect).

1 Like

I expect this to be implemented at some point at least before the end of June 2019. Enketo has some funding for it, but under that special project, it has to be implemented by one of the partner organizations of that project. It's under the High Priority column due to the demand for it: https://github.com/orgs/enketo/projects/2

Most of the work requires no knowledge of Enketo code though (just consuming the API as documented here: https://apidocs.enketo.org/v2). So if somebody else wants to take this on, I would highly encourage that (and we can use this special project funding for other tasks).

The only (fairly minor) work required on the Enketo Express side, is tweaking the authentication (unless ODK Central has in the mean time been updated to support the slower handshake-style authentication that Enketo already supports).

1 Like

before the end of August is the new expectation

6 Likes

please do rely on me for this effort!

4 Likes

They/he/she definitely will. Thank you!

@martijnr do you know how Enketo will track changes made to instance through the server?

With @ln and @yanokwa we are doing quite a bit of work trying to get a GCP compliant audit trail

At the moment we don't worry about server changes to data (because they aren't possible) but if this functionality became available being able to similarly track changes made via the server interface would be excellent and a huge plus on the GCP side.

1 Like

Hi to all, I understand from that discussion that you will make ODK Central able to serve collect forms as web forms. Am I wrong ?
If not this will be a really great new feature !

Mathieu

1 Like

First we'd have add the general audit capabilities to Enketo. I'm open to getting the audit features that ODK Collect has if we can ensure that (meta)data is compatible when collected by different clients (Collect & Enketo). I'm not sure if that is feasible for the complete package of audit events, but for value change logs I imagine it would be since it will be straightforward (not relying on when a field has focus as in the 'question' event). So Enketo may end up with a subset of audit capabilities that ODK Collect has (provided there's funding and developer interest).

Adding the ability to edit already submitted data via Central, could work with whatever audit capability gets added to Enketo, but we'd have to add something to Enketo's edit API endpoint to send the previous audit file with the API call. That is feasible. Once Enketo has the file, loading it would use the same mechanism as loading an unsubmitted draft record (with audit file) from the browser storage.

2 Likes

Yes, it will allow serving ODK XForms as web forms indeed! (like in Ona and Kobotoolbox)!

I'm a little late to the party but just described something similar at Approaches for linking form instance changes to individuals - #22 by LN.

That seems totally fine to me. In fact, I imagine that Enketo might also want to introduce event types that don't make sense from a Collect perspective.

What we might want to do sooner than later is add a way for the client audit log to include a user agent identifier (ODK XForms client type, client version, operating system, etc).

1 Like

Yes, indeed! That would be a good to start early on.

1 Like

since this is a thread about Central after all, i thought i'd chime in and just say that somewhat fine-grained data history (both forms and submissions) and administrative action audit logging are factored into the architecture of Central. i don't think at the interface level it works exactly the way enketo et al do it, but after all it's just architectural foundation right now—exactly what happens upon what action can be decided upon later.

1 Like

Thanks @martijnr.
Such a lot of perspectives ! :smile:

1 Like