We are interested in using Enketo for a form preview to test form appearance and logic, for a PDF of an empty form, and for collecting data in a browser (specifically desktop/laptop data collection/entry, and as an alternative to ODK Collect).
I expect this to be implemented at some point at least before the end of June 2019. Enketo has some funding for it, but under that special project, it has to be implemented by one of the partner organizations of that project. It's under the High Priority column due to the demand for it: https://github.com/orgs/enketo/projects/2
Most of the work requires no knowledge of Enketo code though (just consuming the API as documented here: https://apidocs.enketo.org/v2). So if somebody else wants to take this on, I would highly encourage that (and we can use this special project funding for other tasks).
The only (fairly minor) work required on the Enketo Express side, is tweaking the authentication (unless ODK Central has in the mean time been updated to support the slower handshake-style authentication that Enketo already supports).
@martijnr do you know how Enketo will track changes made to instance through the server?
With @ln and @yanokwa we are doing quite a bit of work trying to get a GCP compliant audit trail
At the moment we don't worry about server changes to data (because they aren't possible) but if this functionality became available being able to similarly track changes made via the server interface would be excellent and a huge plus on the GCP side.
First we'd have add the general audit capabilities to Enketo. I'm open to getting the audit features that ODK Collect has if we can ensure that (meta)data is compatible when collected by different clients (Collect & Enketo). I'm not sure if that is feasible for the complete package of audit events, but for value change logs I imagine it would be since it will be straightforward (not relying on when a field has focus as in the 'question' event). So Enketo may end up with a subset of audit capabilities that ODK Collect has (provided there's funding and developer interest).
Adding the ability to edit already submitted data via Central, could work with whatever audit capability gets added to Enketo, but we'd have to add something to Enketo's edit API endpoint to send the previous audit file with the API call. That is feasible. Once Enketo has the file, loading it would use the same mechanism as loading an unsubmitted draft record (with audit file) from the browser storage.
since this is a thread about Central after all, i thought i'd chime in and just say that somewhat fine-grained data history (both forms and submissions) and administrative action audit logging are factored into the architecture of Central. i don't think at the interface level it works exactly the way enketo et al do it, but after all it's just architectural foundation right now—exactly what happens upon what action can be decided upon later.
I work on KoBoToolbox (one of the partner organizations that @martijnr mentioned) and would like to get some of this done before the end of the month(!) Having read through this thread, it seems like Editing already submitted data probably needs further discussion and won't come together so quickly—but some of what's under the Viewing/pre-viewing/pdf-ing forms and Collecting data in a browser headings could probably be done before September.
i think before we proceed too far we need to work out a couple of things:
what is the proposed method for shipping and operating enketo in our environment? is it optional or always present? how do we manage upgrades?
what is the permissions and authentication methodology between the systems, both at the service level (central can trust this enketo because xyz is true) and at the user level (this instance can be submitted because xyz is true).
i think for these reasons and more, it probably makes the most sense to focus on read-only features first. previewing a form, both interactively and as pdf, and viewing existing single submissions. i think for enketo to actually submit anything we have to really sit down and puzzle through how these features are likely to be used and therefore what sort of permissions/user tracking models make the most sense, and i think those conversations will take some time.
i will warn you that i will be a little bit of a pain in your butt on user experience and on security. i do not hold absolute power over this project but that doesn't mean i can't be loud and annoying. i would really love to see proposed design and criteria (see our own release criteria for v0.6 for an example of our process on this) before work proceeds. some sort of similar proposal process for the technical architecture will help this process a lot.
finally, i will suggest one direction for solving some of these problems: we can enable backchannel privileged communication between services within the docker-compose cluster, and central/enketo can mutually trust any requests made over that communication without authentication. this can be done by exposing a secondary API over a private port that isn't exposed outside the docker network. the asterisk here is that for the purposes of the server audit log (coming in v0.6) it will still be necessary to identify the actor invoking each action.
it may also make sense (or not) to avoid exposing enketo entirely and proxy everything through central.
Right! But my concern would be that this would be too late to be able to use the funding we have now (knowing how busy @jnm gets ;)). Hopefully we can get this fleshed out beforehand.
Do you have a preference for a technical solution @issa? I do not (but also am not familiar with docker). Whatever is best for users to use, install, and update Enketo. It might be useful for scalability if the solution facilitates hosting Central and Enketo on different servers, but maybe that is too complex (for the target user group). I know an important consideration for Central is the cost of running it. It is not really recommended to run Enketo on a droplet with less than 2 Gb of memory ($10), so maybe that means it is best to make Enketo optional.
A. For Enketo to allow only API calls from the trusted Central/KoBo/Ona/Aggregate server, there are 2 levels:
required: an API key provided in the Authorization header with type Basic as the username part (password part is empty)
optional: the Enketo server can be set up to only allow a specific form server domain or domain + path (e.g. only accept the server_url API request parameter value 'https://opendatakit.appspot.com').
These API calls return a webform URL.
B. Submissions to, and obtaining form resources from Aggregate/KoBo/Ona/Central:
Enketo supports 3 methods: https://enketo.org/develop/auth/. I saw a while back that Central supports OpenRosa authentication with a small change (no handshake, if I remember correctly). That would be a minor change we could support in Enketo if (really) necessary. It would be great to use one of these 3 supported methods. KoBo uses the first method. Ona uses the second method.
is there additional documentation on how the external authentication processes work? how does enketo know where the login page ought to be, what does it expect back, etc?
if i'm reading this flowchart correctly, can some of these redirection steps be bypassed if a token is already provided with the initial request?
central has a relatively robust temporary token system that can grant very fine-grained/limited permissions for time-limited tokens, so that could be used here; albeit it sucks to have to do database writes to perform cross-service read operations.
in general, i would be pretty disappointed if we can't work out some kind of relatively seamless/single-sign-on experience between central/enketo, where you never have to understand that you are authenticating to two separate services.
it seems to me that if we can't achieve a fairly tight amount of integration for the eventual end user, we may as well just make sure the services can play well near each other and provide documentation on how to install the two completely separately and then use them together, rather than put a lot of time into interoperation and co-installation only to have the end experience not reflect that investment.
It's a configuration item. See here. Sorry, we are still working on creating more comprehensive documentation for Enketo (same project that funds this work).
I think you're misunderstanding the authentication requirements for the data collector that gets told to use a particular Enketo webform URL to start collecting data. Enketo has no authentication requirements itself for that scenario (and it stores no data). Form access and submission access are only protected by KoBo/Central/Ona/Aggregate if the forms are not public. Whenever Enketo receives a 401 response from Central/KoBo/Ona/Aggregate, when requesting the XForm or form media or external data resources, it redirects or collects credentials (depending on which auth mechanism it is configured to use). (Note that when it uses OpenRosa authentication those credentials are only stored in the user's browser). There is nothing new to be invented here, I think.
The single sign-on experience is supported for the second and third mechanism, but note that this may only be useful for the form administrator, since that is probably the only person that accesses the Central/KoBo/Ona/Aggregate interface.