Pyxform v1.0 release

Thanks, @Ukang_a_Dickson, for leading the way to a v1.0 release of pyxform! I just looked over the project board and I think that if we could finish what is in progress that would be a great release.

The only issue that hasn't been started that I think we may want to consider doing as part of v1.0 is Generate ODK XForms spec-compliant meta block. We've discussed a way of doing it in a backwards-compatible way so it doesn't strictly need to be in a major release but I think it's a big enough change that it would fit well with the release. Of course, there's a question about whether we even consider it important enough to do at all. @martijnr?

@nribeka you have a number of great PRs open. I think that all have some feedback on them for you to address, most very small things. Will you have time to address those in the near future?

I think I can complete most of those reviews. It would be good to get another pair of eyes on support for dynamic defaults and adding a repeat instance in additional to a template when they are ready.

Is there anything else that you think needs to be done before release, @Ukang_a_Dickson?

@LN,

The two PRs is updated based on your latest review. Please have a look and see if I miss anything or if I need to add anything else to the PR.

-Win

Thanks very much, @nribeka! Both look good to me and I have approved them.

I just realized that switching the bind type for selects to string should also be part of this release if we're going to do it. Waiting for feedback from @ggalmazor, @Ukang_a_Dickson, @jnm on whether this impacts other ecosystem tools.

1 Like

Yes, I think so. We should move to following our spec to the letter, and the spec makes sense in that respect. However, fine to do in 1.1 release as well imo.

The version output would be great to get in though (not so much because of 1.0 but because, we'd like to make use of it asap in Enketo).

1 Like

Seems A-OK to me from KoBo's perspective. Thanks for notifying us!

1 Like

I just put up a pull request for supporting last-saved at https://github.com/XLSForm/pyxform/pull/417. It's not strictly needed for a v1 release but it's functionality that we've been getting questions about and a specification that we've had approved for a while. I would really appreciate a review on that.

I gave more thought to Generate ODK XForms spec-compliant meta block and unfortunately I don't feel like it's something I can address. In fact, though I wish we could be entirely spec compliant and I filed the issue, I'd be inclined to mark this one as won't fix because the change seems too disruptive to me. Sorry, @martijnr. At the very least, I don't think it should be blocking this release.

The good news is that the version output requested by @martijnr has been merged as have all of the breaking changes we had planned other than the meta block one.

One thing I would like to suggest is that we add a warning about the change of the root node name from the filename to data. This change is going to be a really great one in the long term but in the short term, it will cause problems for form updates because all node paths will have changed (e.g. /my_cool_form/field will now be /data/field). Here is some example warning text:

If this form is an updated version of an existing form, you may get errors when uploading it to your server. Learn how to fix this at http://xlsform.org#set-root-node-name.

I like that it's non-technical and that it's not alarmist. I don't like that it's very vague and happens at the wrong time since a user won't get an error until they try to upload it. In general, this change is very hard to explain to folks who don't know about the underlying form specification. Do others think it's worth trying to anticipate the problems users will run into? I'd like to try to make this transition as painless as possible. An other idea to help the transition is to continue making the old (current) version of XLSForm online available for some time.

Note that servers which embed pyxform and know what version of the tool was used to generate the existing form version can call pyxform with a value for default_name to keep the root node consistent. They could also suppress whatever warning text we choose to add.

There are a lot of changes in this release and moving to v1 is exciting. I'd really like @Ukang_a_Dickson's blessing before the release goes out. When do you think you might be able to do a pass over the current state of master, @Ukang_a_Dickson?

1 Like

No worries, we can revisit at some point.

So users will be able to manually set the root node name to fix an update issue? That's great. I think it is fine to have this message in pyxform for some transition time, but I think it is best placed on the servers (Ona, KoBo, Central) when uploading (with the link to xlsform.org root node name override) together with other update checks they may be doing. How about taking it out of pyxform in version 1.1?

1 Like

Yes! I don't expect that it would be a problem for systems that embed pyxform, though. Those would be able to directly pass in the desired root node name when a form update is requested.

Upon reflection, I suppose this means that the folks who are most likely to be affected are those who use http://opendatakit.org/xlsform/ or XLSForm Offline to do their conversions. If that sounds right to y'all, then maybe those tools can be the ones to warn. Then we wouldn't need pyxform itself to emit any kind of warning.

1 Like

I'd like to move towards a release by this Thursday. There's a lot of great unreleased functionality that will be very helpful for users and will conclude several roadmap items. I will summarize risks below so that it's hopefully easy for folks to chime in with any concerns. Thank you, @adam.butler for putting pyxform on the agenda for the @TAB call on Wednesday.

Full release notes
  • #357 Add list_name to generated JSON to get static choice list
    • Carlos Quiros @qlands (QLands Technology Consultants)
  • #372 Add support for Java versions that don't have dots in them
  • #289 Do not generate dummy instance content for external instances
  • #378 Fix black configuration for CircleCI
  • #375 Use data as the default root node name instead of the filename
  • #384 If there is only one sheet, use that as the survey sheet
  • #376 Don't generate translation blocks if there is only one language
  • #386 Expand XPath references in upload questions
    • James Tedrick @tedrick (ESRI)
  • #389 Use the filename as the form_id if no explicit form_id is set by settings
  • #392 Add identify-user audit attribute
  • #397 Better Python 2/3 unicode support for ODKValidateError strings
  • #402 Create instance of NullHandler, instead of class reference
    • Brendan Zerr @axonxorz
  • #400 Add track-changes-reasons audit attribute
  • #385 Add support for dynamic defaults
  • #382 Fix crash when displaying ODK Validate errors
  • #380 Add repeat instance in addition to template to the model
  • #408 Remove Python 2.7, 3.5 and 3.6 support, add Python 3.8
  • #412 Preserve attribute ordering across all Python versions
  • #394 Add pyxform version to XForm output
  • #406 Identify first argument of pulldata by splitting on comma
    • Hélène Martin @lognaturel (Nafundi)
  • #410 Output a string bind type for selects
  • #418 Clarify duplicate choice error message
    • Hélène Martin @lognaturel (Nafundi)
  • #417 Add support for last-saved
    • Hélène Martin @lognaturel (Nafundi)

This is considered a v1 release because the XML output will change in several ways. We have done our best to verify that core ODK tools and Enketo will be compatible with these changes, including several versions back. However, older versions of core ODK tools will not be compatible with all changes. This has had negative impact on users in the past (@Batkinson, you may want to make note of these changes). Other downstream tools may be affected, particularly by the following:

  • #289 Do not generate dummy instance content for external instances
    This changes the XML structure.

  • #375 Use data as the default root node name instead of the filename
    This change ensures a predictable root node name. This will make it easier to share raw XPath expressions and will mean that renaming files will not change the XML output. It will also ensure shorter XML output when the filename is long. Users that need a specific root name can introduce the name setting in the settings sheet. For example, when making a permissible update to a form previously uploaded to Aggregate, users will need to use the old filename as the root node name. Tools that use pyxform directly to generate updates to forms will likely want to override the root node name.

  • #385 Add support for dynamic defaults
    Values in the default column that include any of mod, div, *, |, +, -, [, ], {, }, (, ) will be treated as dynamic values and will bet used in setvalue actions in the resulting XML. A warning will be output to let users know that not all tools support dynamic defaults in this way. Tools that don't support setvalue will have unpredictable behavior if used with a form with a dynamic default.

  • #380 Add repeat instance in addition to template to the model
    In Collect, the implication is that one first repeat instance will always be displayed before the user is prompted to add another one. The XML will have changed so some tools may see that as a form update. pyxform will issue a warning.

  • #394 Add pyxform version to XForm output
    Every form generated by pyxform will include an odk:generated-by="pyxform vx.x.x" attribute on the root node name. This will be included in submissions meaning that submissions will be slightly larger in size.

  • #410 Output a string bind type for selects
    Tools that look for select or select1 bind types will no longer work as expected.

  • #417 Add support for last-saved
    This is risky because it extends the way that XPath references are expanded from ${} notation. Additionally, unlike all of the other PRs, it has only been reviewed by folks from one organization (myself and @yanokwa).

4 Likes

The release is now out and XLSForm Online has been updated! Thanks to all who have participated in this big lift.

As I was verifying the release, I noticed another change that I had not explicitly thought about. As described above, the XLSForm filename is no longer used as the root node name. However, it is used as the default form title and form_id but it is no longer validated to ensure that it is a valid XML element name. That means that form_ids with spaces or special characters are now possible.

I have verified that Collect, Central, Aggregate and Enketo can all properly handle form_ids with spaces and special characters. I can't see any limitations on the instance id element described in the ODK XForms spec or in any other documentation so I believe that this is ok but I wanted to flag it in case anyone has concerns.