Form conversion from XLSForm will no longer generate ODK XForms inline selects

In the next release of pyxform, all choice lists will be generated as ODK XForms secondary instances. Prior to this change, forms that did not use any choice filters would have static selects generated instead. You can track this change in the pyxform repository.

I'm tagging this as breaking-change because it does mean some form definitions will change structure when re-converted. This could have implications for analysis or other tools used after data collection. It does not affect any core ODK tools.

We are making this change because there are bugs in when inline vs secondary instance choice lists are generated (pyxform#387) currently resulting in bigger form definitions than needed. It's also very common to use the same choices many times in a form and currently those may be included over and over again.

Additionally, choice lists in secondary instances can be filtered and queried whereas inline choice lists cannot. This has led to hacks like putting true() in choice_filter to force a secondary instance.

ODK Collect and Enketo will continue to support inline choice lists, this change is just for pyxform and what it generates.

To make this concrete, given the following XLSForm survey sheet with a corresponding choices sheet with choices for Apple and Banana:

type name label
select_one fruits fruit Select a fruit

Before:

body:
<select1 ref="/data/fruit">
  <label>Select a fruit</label>
  <item>
    <label>Apple</label>
    <value>apple</value>
  </item>
  <item>
    <label>Banana</label>
    <value>banana</value>
  </item>
</select1>

After:

model:
      <instance id="fruits">
        <root>
          <item>
            <label>Apple</label>
            <name>apple</name>
          </item>
          <item>
            <label>Banana</label>
            <name>banana</name>
          </item>
        </root>
      </instance>

body:
<select1 ref="/data/fruit">
  <label>Select a fruit</label>
  <itemset nodeset="instance('fruits')/root/item">
    <value ref="name"/>
    <label ref="label"/>
  </itemset>
</select1>

We have not received any feedback on this change so have moved ahead and @Lindsay_Stevens_Au is getting close to completing it. This will reduce the size of forms and will resolve a number of bugs.

We do expect some analysis tools will be affected. If you use choice labels in your analysis, please make sure these are either pulled from the XLSForm definition or that your XML parsing code supports secondary instances.

@Florian_May, @Mtyszler I took a quick look at choice label determination in ruODK (here) and I believe that it currently only supports inline selects ("before" above). Does that sounds right? If so, forms after this change would no longer work with the choice label identifying implementation.

Oh what am I talking about, you handled that in https://github.com/ropensci/ruODK/pull/108, @Mtyszler! :tada: :tada: :tada:

Is it possible to retain the old (and broken?) inline choice lists via a commandline option? Or do you really need to strip all this out of pyxform completely? [it will actually break what I'm working on right now, till I finish secondary instances in my iOS parser... :frowning: ]

Any reason you can't use an old version?

Yes, I can use/fork the current version with inline choice support in interim if needed (although I do love https://getodk.org/xlsform/ :grin:).

1 Like

Sorry to hear this will affect you. I posted the user-facing wins above but another big motivator is that it lets us remove a bunch of logic that’s not tested and focus all select tests on one type of output. Most of that code is already out so at this point we really do want to go with the full change.

We can commit to holding it for one more Central release. That means it would be released in 2023.4 which would be out in September at earliest.

1 Like

I'll shout the Core team a round in London :slight_smile:
[BTW, totally understand the desire to spring clean :+1:]

Agree. It has taken a while to get this far switching over to itemsets (and we haven't even started code review!). Supporting both new and old code paths for an itemsets flag would be significant extra work, and complicate future maintenance.

1 Like

This change is now merged. In doing quality assurance, we have identified that it surfaces a number of Enketo bugs. In particular, there is at least one related to repeats that we haven't identified the root cause of yet.

The change is available at https://getodk.org/xlsform/. Please try it out with your forms and do let us know if any of them result in errors or unexpected behavior. We don't think errors will be common but the more information we get, the faster we can solve the problem.

1 Like

We have fixed all the issues that we are aware of. Thanks to everyone who has tried it out on https://getodk.org/xlsform/!

We will release this change and a few others as pyxform v2.0.0 next week. This will be included in the next Central release which should be out in the next two weeks.

Consider trying out your forms and making sure the convert as expected before then. If we hear of other issues, we will hold the pyxform release until they are fixed.

You can see the changes from this release at https://github.com/XLSForm/pyxform/compare/v1.12.1...master

1 Like