Instance lookups on internal choices / Instance logic order with Enketo

ahblake · August 12, 2023, 11:49am

These are issues I've been working around for a while but finally managed to create an example form for and document.

1. What is the issue? Please be detailed.

When using an instance lookup, it doesn't work for internal choices if structured the same way it would be for an external CSV. (This is a known issue, posting mostly for reference)
1.1 With a file CSV_choices.csv that has a column emoji, this works as a calculate: instance('CSV_choices')/root/item[name=${CSV_choice}]/emoji
1.2 WIth an internal choice list (list_name = internal_choices) also with a column emoji, this fails validation at upload: instance('internal_choices')/root/item[name=${internal_choice}]/emoji. Putting any choice filter on the select for the field used in it will allow it to upload and work per this note in the docs and this issue

The XLSForm could not be converted: ODK Validate Errors: >> Something broke the parser. See above for a hint. Error evaluating field 'calc_internal' (${test_instance}[1]/calc_internal[1]): The problem was located in Calculate expression for ${calc_internal} XPath evaluation: Instance referenced by instance(internal_choices)/root/item/emoji does not exist Caused by: org.javarosa.xpath.XPathMissingInstanceException: The problem was located in Calculate expression for ${calc_internal} XPath evaluation: Instance referenced by instance(internal_choices)/root/item/emoji does not exist ... 10 more The following files failed validation: ${test_instance_lookup}.xml Result: Invalid

When using an instance lookup with filters beyond just one/many value = ${field} statements, the order of the value and field matters to Enketo
2.1 This calculate has value, field order then a contains() and has no issues. count(instance('CSV_choices')/root/item[shape = ${shape_type} and contains(${CSV_choice_mult},name)]/name)
2.2 This calculate has field, value order and has no issues. count(instance('CSV_choices')/root/item[${shape_type} = shape]/name)
2.3 This calculate has field, value order then a contains() and (a) throws an error when opening the form in Enketo, (b) breaks the selects elsewhere in the form & (c) doesn't calculate. It functions as expected in Collect though. count(instance('CSV_choices')/root/item[${shape_type} = shape and contains(${CSV_choice_mult},name)]/name).

Error occured during the loading of this form. We do not recommend you use this form for data entry until this is resolved.
Please contact support@getodk.org with the link to this page and the error message below:
FormLogicError: Could not evaluate: count(/model/instance[@id="CSV_choices"]/root/item[ /model/instance[1]/data/test_instance_order/shape_type = shape and contains( /model/instance[1]/data/test_instance_order/CSV_choice_mult ,name)]/name), message: callNative() can't handle nodeset functions yet for shapeandcontains()

With calculate in 2.1 above, no suppression (is the spacing difference related to enketo core #988?):

Choices for widget using select_one_from_file suppressed & `columns` appearance for unrelated question above suppressed and note text bolded due to calculate in 2.3 above:

2. What steps can we take to reproduce this issue?
Try attached form, reintroduce breaking calculations

3. What have you tried to fix the issue?
searched forum/read docs

4. Upload any forms or screenshots you can share publicly below.
test_instance_lookup.xlsx (34.4 KB)
CSV_choices.csv (165 Bytes)

LN · August 13, 2023, 12:14pm

As you may have seen in pyxform and at Form conversion from XLSForm will no longer generate ODK XForms inline selects, we recently merged a change to always create secondary instances for selects to support exactly this usage. It also makes pyxform code simpler and makes form definitions smaller.

Unfortunately, it has led us to identify a number of issues in Enketo ranging in severity from cosmetic (the issue you linked below) to likely serious. We're still working on how to approach these unexpected findings.

This looks like a new issue.

ahblake · August 24, 2023, 2:03am

New issue! I was very close to literally tearing some hair out with this until I though to test not looking up the image/big-image filename. (currently working around it by using translate and concat on the name value to build the image filename for the dynamic image annotation, I could also add another image column, six of one...)

These don't work, even with a choice filter on internal_choices
instance('internal_choices')/root/item[name=${internal_choice}]/image
instance('internal_choices')/root/item[name=${internal_choice}]/big-image
instance('internal_choices')/root/item[name=${internal_choice}]/label

If I rename the column big-image to bog-image and update the calculate, it works as expected. Other columns also look up fine. These are the only ones I've identified with failing lookups.

Have I missed something elementary here or is this a real bug?

LN · August 24, 2023, 10:45pm

I don't believe the image/big-image filename is currently intended to be dynamic at all. That is, putting instance('internal_choices')/root/item[name=${internal_choice}]/image in the image column means clients are happily looking for a file with literal name instance('internal_choices')/root/item[name=${internal_choice}]/image and not finding one.

A default value including a dynamically-generated one can be applied to a question type that displays an image. When that question loads, it will show the image represented by the dynamically-determined value.

image/big-image are types of labels and there is currently no mechanism for those to be dynamic (but it is indeed on the very long todo list, especially for the entity context).

ahblake · August 24, 2023, 11:57pm

Sorry, i wasn't clear.

I am trying to instance lookup the filename into a calculate (which i then use later on in a calculate for an image) and while i can calculate other values into that field with no issues, the image, big-image columns won't.

I'll create an example form later to demo

LN · August 25, 2023, 9:47pm

I think I get what you're saying! Many XLSForm columns are used as dynamic values. That means any expression you write in them will be evaluated and their result will be used. However, image and big-image take static values. They take whatever expression you write in them and use that as text. Doesn't Central prompt you to attach an image with filename instance('internal_choices')/root/item[name=${internal_choice}]/image?

If that doesn't sound right, an example would be helpful, thanks.

ahblake · August 25, 2023, 10:56pm

@LN I should have spent more time making an example than trying to explain it, as clearly I haven't been able to do so!

From my choices list, I want to extract the values in image and big-image, and then use one of those when building the jr://image/EXTRACTEDVALUE string that is used in a calculate for a dynamic default image. If the image filename is stored in a column that is NOT the image/big-image column, it's fine.

Additionally, the lookup for the label doesn't work either, but you can use jr:choice-name for that. I thought that perhaps jr:image might work, but it doesn't, unless it's jr:image-name etc.

See the last group in this form;

test_instance_lookup.xlsx (35.9 KB)

LN · August 25, 2023, 11:49pm

Ahh, I'm with you now! I didn't catch that you were talking about the choices sheet.

This is because image, big-image and label have special meaning to the XLSForm converter when used on the choices tab. Here's the XML that's generated for the "Pomme" choice:

<text id="internal_choices-0">
  <value>Pomme</value>
  <value form="big-image">jr://images/Image1.jpg</value>
  <value form="image">jr://images/Image1_SM.jpg</value>
</text>
...
<item>
  <itextId>internal_choices-0</itextId>
  <emoji>🍎</emoji>
  <name>apple</name>
  <notanimage>notanimage1.jpg</notanimage>
  <shape>round</shape>
</item>

You can see that under the hood choice images use the same mechanism as translations. If you remove the image and big-image columns, the following XML will be generated for "Pomme":

<item>
  <label>Pomme</label>
  <emoji>🍎</emoji>
  <name>apple</name>
  <notanimage>notanimage1.jpg</notanimage>
  <shape>round</shape>
</item>

The itext/translation indirection will be removed and you'll be able to get the label directly.

This is implicit and very annoying, I'm sorry you've run into it. I'm not sure what we can really do about it without rethinking the way translations and images are defined, though.

ahblake · August 26, 2023, 5:12am

Now that i know it's not me doing something wrong, i know that all i need to do is duplicate the filename i want in another column (eg defaultimage) and calculate that in.

wroos · September 16, 2023, 12:12pm

@ahblake
Would you mind to share the XLSForm (extract) for your final solution.

wroos · September 16, 2023, 12:23pm

Hi @LN

Could we get a list (preferably in the documentation) which columns are static, please?

Furthermore, there seem to be columns which are dynamic, but only cover limited expressions, e.g. label (no if construct, sum etc.). Which ones?

LN · September 19, 2023, 5:33pm

I believe the ODK docs always explicitly state when expressions can be specified in an XLSForm columns. All values specified elsewhere are treated as static.

label and the fact that it can include references to other fields is described in the XLSForm overview. Can you think of another good place for that information?

Instance lookups on internal choices / Instance logic order with Enketo

With calculate in 2.1 above, no suppression (is the spacing difference related to enketo core #988?):

Choices for widget using select_one_from_file suppressed & columns appearance for unrelated question above suppressed and note text bolded due to calculate in 2.3 above:

Choices for widget using select_one_from_file suppressed & `columns` appearance for unrelated question above suppressed and note text bolded due to calculate in 2.3 above: