Custom form metadata

[tomsmyth]

What is the general goal of the feature?

The ability to set arbitrary key-value pairs under the Form Metadata area in General Settings, and then to access those values in forms via jr:preload or meta elements.

These custom values should be included in the 2D barcode.

What are some example use cases for this feature?

Our client wants to be able to set a Team Number field on devices, with that number being included in a unique identifier that gets generated for a given survey respondent. They want to avoid having the user entering the number every time to improve data quality.

There is already the Username field and we could use that (and probably will for now), but it feels a bit hacky.

Furthermore, now that the three 'user defined' fields have been decoupled from Server Settings (great!), they all of a sudden seem a bit arbitrary. Why are we privileging these three data types (username, phone, email)? The answer I think is historical, but I suggest we could transform that section to be fully user definable, perhaps with the existing three fields as 'suggestions' that you could delete if you don't want to use them.

We could also perhaps change the labels to the actual keys (userID, phoneNumber, email) and enforce some format restrictions on keys so that new user-defined ones use that style.

And it would also be nice to allow using the camelCase meta tag names (userID, simSerial, phoneNumber etc.) as jr:preloadParams. We'd have to still support the old ones obviously, but is there a reason this can't be added? Maybe even allow instanceID, timeStart, and timeEnd as values for jr:preloadParams with jr:preload set to property and slowly deprecate the jr:preload field altogether? I don't understand why the two of them exist separately.

If this were in place, I think it might solve a bunch of needs folks might have around carrying data across form instances. e.g. Imagine an enumerator is going to survey 50 houses in a given town today, then 50 in another town tomorrow, they could set the town name in Metadata at the start of the day instead of entering it every time.

Another idea that has been floated is remembering the last answer and setting it as the default. I think this would also be a useful tool in the toolkit. We also needed this feature and ended up using an external app (thanks to @Grzesiek2010) as there was a fair amount of pushback on building that feature into ODK for reasons I can't remember. Also in that case it is numerical data, and we are were doing stuff like incrementing the number, so an external app seems more justifiable. But if it's just remembering the last answer, that seems like overkill. So I'd still like to see this make it in some day too.

But I think both approaches (remember last answer and custom metadata) would be useful depending on the situation. Imagine for instance if you have a team that is pre-configuring devices for folks before they head out. They could setup certain metadata values and verify them to be correct, and the enumerators might not ever need to touch them. This could be important for data quality.

The other consideration is that we already have a mechanism and a clear precedent for preloading form metadata. All we'd be doing here is extending the XForms spec a bit to allow arbitrary key value pairs.

Finally, if we do this, It might also be nice to move 'Form Metadata' to the top level of settings to allow easier access. There is plenty of room on that screen now

What can you contribute to making this feature a reality?

Probably a good amount. But first need to hear if others think this makes sense.

Thanks for listening!

[martijnr]

Thanks for this proposal Tom. I believe your proposal includes 2 things:

1. add custom metadata properties that can be set on the data collection client
2. add an automatic preload item that obtains the last entered answer from a previous record collected with the same form.

I didn’t add “changing preloader syntax” as it seems this could be seen as an implementation detail of #1 and/or #2.

In my opinion #1 seems like a good idea for the ODK XForm spec. I'm not excited about #2 myself and think it is more suitable for a port of ODK Collect (or Enketo). It seems too particular to include in our spec. However, a solution for #1 may provide a straightforward path to implement #2 in a port.

An @ln mentioned in the latest dev meeting, an artificial secondary external instance seems like a great way of exposing the custom metadata so it can be accessed by the “form”. Commcare’s session instance seems to already cater to this. If we would implement that we could add the custom metadata under the <data> block. A logical consequence would be to also use the same session instance to expose and obtain the existing fixed metadata (deviceid, username etc).

If we would implement just this special session instance without anything else you could already meet your requirement #1. By adding a calculated item like this:

<bind nodeset=“orx:meta/orx:customProp” calculation=“jr://instance/session/data/customProp”>

This bind would populate the field upon load. However, as you proposed it is probably a good idea to look into using the same preload mechanism (or a replacement/improvement of this) to populate the custom metadata. That’s potentially a big discussion, so I would like to leave it at this for now and continue later.

[tomsmyth]

Martijn, you are correct. My intent for this thread was to propose #1, but I also mentioned #2, which lives at this thread.

I'm glad you like the idea of #1.

I'm sad you don't like #2.

Based on the other thread it seems like folks could benefit from it. I'm still not sure what the hangup is with this kind of thing, but I personally am not going to push it since the custom metadata (subject of this thread) would suit us fine.

As to your implementation suggestions, they seem fine to me.

Our users never see the XML so whether we are doing orx:meta whatever or jr:preload whatever, it doesn't matter as long as it works.

I also care about the purity and harmony of the ODK xforms spec, of course, so I'm happy to contribute to that discussion.

[LN]

Taking a quick step back from the implementation for a moment, I want to make sure we're on the same page when it comes to user experience. Clients would implement an interface that would let users set arbitrary key-value pairs. The details would be up to the client implementation but this JSFiddle shows roughly what I have in mind. Here's a screenshot:

There would be some validation to make sure that the keys are valid identifiers according to whatever backing scheme we implement, that they're unique, etc and the values would be freeform. Are we on the same page there?

Artificial Secondary External Instance Implementation
I haven't had the opportunity to think deeply about artificial secondary external instances yet. I believe the concept is that backing data can be anything (Shared Preference key-value pairs in Android for example), but that it is accessed in a standardized way (XPath). Does that sound right, @martijnr? For example, the virtual XML equivalent to my screenshot above would be something like:

<session>
	...
    <data>
    	<team>purple</team>
    	<favorite-food>lasagna</favorite-food>
    	<todays-code>7756</todays-code>
    </data>
    ...
</session>

But it doesn't ever really have to be XML. Since custom values would have no children and no known names, there's no real meaningful XPath queries to write over the virtual instance anyway. Am I getting this right?

Is data the right parent node name or might there be advantages to going even more specific with something like client-custom-values? Or perhaps that would be a child of data?

jr:preload Implementation
jr:preload is a little unfortunate because it defines new syntax where using an artificial secondary external instance wouldn't have. However, the mechanism exists. Couldn't another option be to say that jr:preloadParams can have arbitrary text and will fetch a custom value if available. For example, <bind nodeset="orx:meta/orx:team" type="string" jr:preload="property" jr:preloadParams="team"/> would allow me to populate the orx:meta/orx:team node with the value I specified in my JSFiddle above.

@tomsmyth

I also care about the purity and harmony of the ODK xforms spec, of course, so I’m happy to contribute to that discussion.

Likewise. So perhaps this isn't a great approach but it does have the advantage of being consistent with past decisions?

XLSForm interface
I also want to give some thought to what XLSForm users experience. This will hopefully also give some ideas for other form builders. The best I'm coming up with currently is to build on the idea of artificial instances by allowing some prefixes in the ${} syntax. So for example, I would do something like

+-----------+---------+-------+-------------------------------+
|   type    |  name   | label |          calculation          |
+-----------+---------+-------+-------------------------------+
| calculate | my-team |       | ${/client-custom-values/team} |
+-----------+---------+-------+-------------------------------+

We could make the prefix anything that is user friendly, in this case I used /client-custom-values/ as an example but it doesn't necessarily have to be related to the XForms approach.

I think that in the case of a jr:preload-based implementation, an intermediary calculate as described above would be necessary. But in the case of an implementation based on artificial secondary instances, it might be possible to use the value directly as in

+-------------------+-------------------+---------------------------------------------------------+
|       type        |       name        |                          label                          |
+-------------------+-------------------+---------------------------------------------------------+
| select_one yes_no | team-confirmation | Are you part of the 
|                   |                   | ${/client-custom-values/team} team? |
+-------------------+-------------------+---------------------------------------------------------+

I'm imagining this would translate to

<bind nodeset="/my-form/team-confirmation" type="select1"/>
...
<select1 ref="/my-form/team-confirmation">
  <label>Are you part of the <output value="jr://instance/session/data/team"/></label>
...

I'm less sure about this -- it's not totally clear to me what's allowed with output.

[tomsmyth]

This all sounds grand to me. I think you've captured my request accurately.

We don't have an iron in the XLSForm fire, but another option might be to introduce a new question "type" like metadata and then have a new column called like "metadata::key" for specifying the key? You could then load both the existing metadata in that style or the new custom metadata ones.