XLSForm spec proposal: add syntax to make it easy to use a value from the last saved instance

I think your broadened concept always requires specifying a record in some way, right? That is, to query arbitrary record you need to specify both the field you want and from which record you want it. I see how a short syntax like !{} can specify a specific record (e.g. the last one) but it's not clear to me how that could be generalized. Maybe you could share an example?

You're right, it broadens the potential scope for this new handle based on features we haven't built yet, i.e. case management, so it's probably premature to try and consider this use case. I suppose examples would be

recordID/data{street} or data{recordID/street}

in which case recordID could be the UUID or a different unique lookup field that specifies the record from which data should be retrieved. If and when we support case management, this would be a potential way of implementing this using the default column. It would be nice if the new syntax we're introducing now can logically be extended to accommodate this new feature. But of course it could do that also with a syntax of !{street}.

1 Like

I like the idea of consistent syntax between representing a value from the last saved instance and one from any instance as @Tino_Kreutzer is describing but I don't feel confident that I know enough about what the general case will look like to design for it.

My sense is that the ID of a record that needs to be consulted would always be dynamic (so ${recordID}) which makes it hard to use as a prefix and that the goal would be to fetch records based on many different characteristics, not just ID. I'd guess that advanced users will use XPath directly (as they already do) and there will be convenience functions for those who don't need the full flexibility (like pulldata or indexed-repeat).

I have a slight preference for something that leverages the existing ${} syntax because it feels like, to use @adam.butler's language, it's the same interpolation but with a qualifier:

  • ${street} goes to /data/street
  • whatever new syntax is agreed on goes to instance('__last-saved')/data/street

I agree that __ is hard to deal with in a user-facing context so let's take ${street__last} off the table. To answer @tomsmyth's question, only characters that are valid in XML element names are allowed in field names: https://www.xml.com/pub/a/2001/07/25/namingparts.html. If we do stay within ${}, then, we can use a separator that is not valid in XML to make absolutely sure it can't conflict with a user-given name. Something like ${street#last-saved} or ${last-saved#street}.

Introducing a whole new thing like !{} or #{} feels a bit heavy for something that probably won't be used in so many forms. I also think it's hard to remember which special character to use. I'm not deeply against it, though.

I'm not thrilled about something that looks like a function but isn't but I could be convinced.

Yes this is my thought exactly. It's quite like the psuedoclass concept in CSS. What about ${street|last}.

1 Like

My experience is that non-developers can spend a LONG time hunting for the pipe character on their keyboards and/or use a capital i or a lowercase L and get very confused. It's especially confusing because some keyboards have it labeled as a split pipe (¦).

An additional requirement: whatever characters need to be typed should be recognizable to anyone.

That's a good point about the split pipe, @LN - I also think that # is a good option. I've been thinking about @Tino_Kreutzer's idea of thinking forward to possible case management use cases. It seems like it would be helpful to come up with a generic way of saying "the value of field x of entity y". In this particular example, x = "street" and y = "the last saved form". In a possible case management scenario, we would probably want to reference fields on the root entity (where "root entity" means e.g. "the patient I'm reporting on", or "the tree that I return to measure every month") (NB I'm not saying that "root entity" is the best name for this, just using it for the sake of these examples!).

In the case management scenario, we would probably want to reference these fields in labels (e.g. "what is 's temperature?") or skip logic (e.g. "skip the next question if the tree is partially in shade"), rather than default values, so that's worth bearing in mind: it would be good to come up with a syntax that is also usable in both of those contexts.

It seems like we all agree on ${...} as being a reasonable way of representing "the value of", so now we just need to work out the best way to say "field x of entity y". The two options we've talked about are x#y and y(x), but I wonder whether it might also be worth considering something along the lines of y#x, as @LN suggested?

Possible values of y would be pre-defined, such as last-saved or root-entity (or more succinctly, last and entity). So in the examples that we have:

  1. Last filled street: ${street#last} or ${last#street} (it might be worth allowing an abbreviation if the field name being accessed is the same as the current field name, so then it would be ${#last} or ${last#}
  2. Patient name in label: what is ${full_name#entity}'s temperature? or what is ${entity#full_name}'s temperature?
  3. Tree status in skip logic: ${partial_shade#entity}=yes or ${entity#partial_shade}=yes

I think my vote would go to ${y#x}, i.e. ${last#street} / ${entity#full_name} / ${entity#partial_shade}

1 Like

@adam.butler I like where you're going.

Here's what I'm understanding for ${entity#full_name}:

  • there'd be some kind of standard identifier (e.g. recordId) linking the current form instance to info previously collected about the entity this form instance is concerned with. Presumably, the previously-collected info would be in an external secondary instance representing all entities and their info.
  • the XForm would define a __entities instance to give access to all of the entities' info
  • entity in ${entity#full_name} would expand to something like instance('__entities')/recordId/data
  • the entity# shortcut would only allow referring to values related to the entity this form is about, not other entities. For example, if I'm defining a form that will collect information about houses, I can use it to refer to information previously collected about a specific house but I can't use it to refer to information about the neighbor's house or an occupant of the house that info is being collected about.

Did I get that right?

I think it's a useful concept even with the limitations in my last bullet above. I'm on board for introducing ${last#<unqualified field name>} for now with the goal of expanding to other prefixed keywords like ${entity#<unqualified field name>}.

Yes, yes, yes and yes :slight_smile: - and thank you for making all of that explicit @LN !

Just as an addendum to your last bullet: if the occupants were somehow marked as being a direct attribute of the house entity, then I think that it ought to be possible to refer to them using this syntax. But that's probably a discussion for another day...

1 Like

A possible issue I see with this definition of #entity is that it appears (correct me if I’m wrong...) to tie entities to a specific instantiation of a specific form id+version (?). Whereas in the general case - and certainly in mine, you can perform multiple (and completely different) ‘inspections’ (aka fill in completely different forms) about the same ‘entity’, so this vague “entity” thingy exists independent of any particular form, yet alone specific form version.

In this context, What is an ‘entity’? Or should it instead be called say “specific form instance”?

“Entity” to me conveys a physically unique object. Whereas a form instance/submission is rather more an partial snapshot in time, unique only unto itself.

@Xiphware the way I see case management working in ODK is that you would have two different types of form: entity forms and report forms (this nomenclature is not yet written in stone). There's more information about the proposed approach here:

It's due for another round of TSC discussion in the near future.

Thanks, that makes more sense. This is actually pretty much exactly what I do in GoMobile, except we call these 'entity' forms the "Asset Form" associated with an inspection. This basically holds/displays/captures the (mostly) static data about the thing you are inspecting; eg building address, owner details, property id, etc. This model is working quite well for us in the field - every scheduled inspection also downloads the associated asset form, so all the fixed asset data (typically from the council's backend database) is available to the inspector when out in the field, and possible offline.

Indeed, one of the features I'll be implementing is adding the ability to pull in arbitrary asset data into an inspection form (for whatever reason), so I'm acutely interested in how this might be (naturally?) accomplished by being able to reference the asset data associated with an inspection via a suitably identified external instance/XPath query.

1 Like

EDITED to use last-saved instead of last as the prefix.

How have the latest ideas aged through various showers/sleeps/brain marination sessions?

Here is what I think the latest proposal looks like:

  • If the contents of the default column are not a literal value, the expression is passed through to the value attribute of the setvalue action (see documentation) triggered on first load (using the odk-instance-first-load event). Literal values get treated the same way they do now.
  • Introduce new syntax to refer to a question in the last instance: ${last-saved#<question_name>}
    • If an XLSForm definition uses this construct, <instance id="__last-saved" src="jr://instance/last-saved" /> will be added to the XForm
    • ${last-saved#<question_name>} will expand to instance('__last-saved')/data/question_name (with additional group/repeat levels as needed -- same as with the ${<question_name>} construct).
survey type name label default
text street Street ${last-saved#street}
date disaster_date Disaster date today()
integer patient_count How many patients have you seen today? if(${last-saved#patient_count} == '', 0, ${last-saved#patient_count} + 1)
select_one yes_no same_street Are you still on ${last-saved#street}?



Yes, this looks good to me. Since we're introducing some major new syntax here, I'd like to make double triple sure that (a) this is consistent with any other similar syntaxes that already exist, and (b) it's flexible enough to cover a reasonable number of similar future requirements.

As far as I can tell, the answer to both questions is "yes", so :+1: from me.


Agreed, @adam.butler. Would the TSC perhaps be willing to put this on the next agenda to discuss for 5 minutes and make sure we have considered every angle and that there is consensus to proceed? This has felt like a very good discussion with most TSC members involved but you're right that since it could be potentially far-reaching, we should make sure everyone has had time to consider it carefully.


Wow, great discussion! I want to bring up one potential issue that I think hasn't been mentioned before, that may impact the design (if we cannot solve it in the pyxform implementation).

On the XForm side, there is a subtle difference between using <setvalue> with the odk-instance-first-load event and the current defaults (default values in instance) wrt to repeats. If a new repeat is created the current default (value in instance) would populate the default value. With dynamic defaults it wouldn't if the odk-instance-first-load event would be used (because it happens after form loading). I'm thinking introducing a new repeat-creation event with pyxform coding magic may resolve that (CommCare has jr-insert).


Overall, this looks great to me. Two nits to pick.

  1. The use of two underscores to prevent naming collisions is fine (seems to work for Python), but using it implicitly sets a standard for how pyxform generates these IDs. Are we sure this is the best syntax?

  2. I wonder if last is verbose enough. I can imagine a form designer thinking that it's the data from the last visit rather than the last saved. Why not just call it last-saved.

Thank you, @martijnr! That's an interesting case I certainly had not considered. Agreed it should be supported and adding a repeat-creation event seems like the best way to do it. There are already various cases in pyxform where the output is different based on nesting in a repeat so I don't think it would be too terribly big of an addition.

We should probably have a new thread for introducing and naming the event, right? JavaRosa/Collect does also support jr-insert even though it's not documented. I'm not sure how it differs from xforms-insert.

Good point. This refers to __last-saved in <instance id="__last-saved" src="jr://instance/last-saved" />. I don't think there is a case currently where pyxform has to generate an identifier. Can you think of any, @martijnr or @Ukang_a_Dickson? If there is, we should match that convention. Either way, that convention should be documented so it can be reused as needed.

I'd be happy with that.

1 Like

Can't contribute to the technical conversation but I work for MSF (Medecins Sans Frontieres) and we have implemented an ODK-based hand hygiene compliance observation tool across multiple health facilities in 15+ countries. This feature was on our priority list for future versions so observers didnt't have to keep filling in their name, location, ward, etc (in fact, discussed with @zestyping who pointed me at this thread).

So, thanks from us for taking it forward!


Hi, Pete! Great to see you here. This is the right place to be—it'll all get worked out on this thread; I'm guessing there will be more discussion in the coming weeks.

1 Like

@pedrito1414 if you have feedback on the XLSForm structure as shown in the sample spreadsheet in this post from a user perspective, that would be very helpful! Is it intuitive? Does it serve the needs you have in mind?