XLSForm spec: also resolve ${foo} names in custom settings attributes

(or perhaps this is just a feature request for pyxform? or perhaps this is just an existing bug in pyxform?...)

What high-level problem are you trying to solve?

Basically the desire here is to resolve form question identifiers - eg ${foo} - to their respective XML instance element names - eg /data/foo - when adding custom instance attributes in the settings sheet, much as they are already resolved elsewhere in XLSForms.

Any ideas on how ODK could help you solve it?

Currently, when writing XLSForms, if you wish to reference specific questions (or calculations) in the form you can use the shorthand notation ${foo} to reference the specific question whose name=foo. This is pretty much universal behavior in XLSForms.

Further, when you add custom instance, binding or body attributes to questions - as described in the XLSForm spec - if you put ${foo} as the value for the attribute then it is correctly resolved to the appropriate XML element description in the resulting XForm definition. For example, the following XLSForm has two questions and I've added three custom attributes to each:

the resulting XForm looks like:

As you can see, the XLSForm ${name} reference has been correctly resolved to /data/name in the custom isParent instance, binding and body attributes in the final XForm.

However, for similar custom attributes against the main instance element, which are specified on the settings sheet using a column(s) with an "attribute::" prefix, these XLSForm ${foo} style references are not resolved (!); instead, the raw string is simply copied into the resulting XForm:

As written, the XLSForm specification appears to be non-specific about resolving ${foo} type references; however, other than apparently this one case with custom root instance attributes, it does appear to be almost universal behavior.

Summary

The desire here is to be able to reference form questions using the regular ${foo} XLSForm shorthand when setting custom XForm root instance attributes, just as you can for setting other custom XForm attributes in the primary instance, bindings, or body.

Its not clear if this was a deliberate decision not to [if so then I'd like to understand the rationale], or an oversight when implementing pyxform support for custom attributes by not including root instance attributes in scope, or pyxform should already be doing this and it is a bug.

Workaround

Note, you can manually resolve and enter the correct XML element name directly in the value instead, eg "/data/name", but this is awkward and atypical, and relies on the form builder being cognizant of the XML root element name [this is almost always "data", but it can vary...]. So it is much preferable to be able to specify ${foo} in this context as well.

2 Likes

This does feel more like an accidental omission than anything else. I think a PR for this would be welcome. @Lindsay_Stevens_Au does that sound right?

1 Like

Minor suggestion, it may also be worth explicitly documenting (in XLSForms) which of these custom settings may be calculations vs XML references vs static strings only. Although it becomes obvious in the resulting XForm definition, its not immediately apparent why, say, instance_name is actually evaluated as an XPath expression, whereas name must be a static string (not a ref!), whereas custom bind attributes may be static string or a ${foo} ref.

Seems like a feature request, rather than a bug or oversight:

  • Ability to specify custom attributes on the main instance element was added in 2015 (here) but the discussion did not mention resolving XPaths as part of the design/requirements.
  • Tests (old, new) only checks use with static strings.
  • Documentation on XLSForm docs (here, but not in the XLSForm template) only mentions use with static strings.

Questions for @xiphware:

  • If you manually add the desired XPath reference to the XForm, do Collect and Enketo insert the value into the attribute in the form instance/submission XML? i.e. is value resolution of custom attributes on the main instance element supported by clients
  • Could you please share a concrete example of a use case for this feature, and what workflow(s) this feature would enable for you?

Questions for @LN / @Aly_Blenkin:

  • Based on your experience with custom attributes (or knowledge or user workflows), would you expect that anyone would want this resolution to not happen? i.e. what's the risk that adding this feature would break someone forms/workflows.

Actual changes needed (here) would be pretty minimal.

No, the value of the referenced XML element is not inserted, but that is not actually the desire. Rather it is simply the ability to specify ${foo} in XLSForm and have, say, /data/foo appear as the string value of the attribute in the resulting XForm XML, and have "/data/foo" sent up in the POST submission payload.

So by 'resolve' above I just mean resolving ${foo} to the XPath reference path. I guess you could potentially go a step further, and actually lookup that XPath element (eg at final form validation step?) and insert its value into the attribute, but that seems like a further step with a lot more potential gotchas... And most definitely a feature request! :slight_smile:

But to answer your question, if you do put say "/data/foo" as the value of attribute::customSettingsAttribute1 in your settings, only "data/foo" is sent back in the submission, as expected. This is consistent with the behavior of how custom body::, bind:: and ::instance attributes work today (see above example), and I dont think we should change it.

Could you please share a concrete example of a use case for this feature, and what workflow(s) this feature would enable for you?

Our specific usecase if that we wish to add a custom field into forms for our workflow that our backend (Kobo) knows about to extract from any submissions coming in. Specifically, in our immediate need is to add an (optional) summary status message that our backend will echo back to the client upon submission, by populating the (optional) OpenRosaResponse with it (see https://bitbucket.org/javarosa/javarosa/wiki/FormSubmissionAPI). That is, the form as part of its calculations determines the (possibly hidden) summary outcome, submits the results back, and the client - Enketo in this case - picks this up from the OpenRosaResponse message response and displays it instead of a default "Thank you for participating". Obviously, some clients may already completely ignore this OpenRosaResponse message, so it wont break anything.

Note, this could, alternatively, be accomplished by introducing, say, a new metadata field (eg "meta/responseMessage") but that involves a much greater spec/ecosystem change IMO. Whereas it seems these extensible custom attributes on the instance (https://xlsform.org/en/#advanced-use-and-extensibility) seem ideally suited to this purpose. However, in order to be able to dynamically provide a custom return value (eg the result of a previous form calculation) our attribute needs to reference a form-specific XML element. [if we instead made the form-specific XML element a universal 'magic field' to be used by all forms to return a response message, we're ostensibly back to adding and documenting a new metadata field...]

This usecase can already be accomplished today but it requires manually putting the correct XPath ref in the attribute value, which requires the form designer knowing the root element is usually called "data" - eg "data/myResponse" - whereas it would be cleaner, and IMO more consistent, for them just to be able to put ${myResponse} as the custom attribute value and have pyxform resolve it.

Hopefully that helps explain a bit cleaner. Again, the desire isnt to actually lookup the custom attribute value from the referenced XML element at runtime, just to resolve ${foo} to "/data/foo" during pyxform compilation time.

1 Like

BTW this is certainly an interesting idea and would make the job of the backend much easier - it could then extract the desired field value directly out of the POST submission's root instance attributes - rather than the BE having to extract the path first and then lookup that ref in the XML payload before responding to the incoming submit POST... But doing this upfront would require (non-trivial?) functional changes to both the Collect and Enketo clients. Whereas the proposal here is just a pyxform 'tweak'... :slight_smile: [and obviously some BE work for Kobo]

FYI I've created a new pyxform github ticket to track this issue here: https://github.com/XLSForm/pyxform/issues/717