Current() in XPath predicate

So just to be clear, the "fix" we're testing here reverts the entire
context within that populateDynamicChoices() function, to be the field
instead of the parent -- so the effects are potentially much wider than
just what current() returns. Potentially anything and everything that
depends on context can change now, since the context will shift from parent
to field.

This means that we revert a change from 5 years ago and alter the context
from now on. Neither Meletis nor I feel terribly confident that we can
entirely predict the consequences to backward compatibility. We're testing
the things we can think to test, but honestly it's XML users in the broader
ODK community who face the greatest risks here; SurveyCTO users by and
large use XLSForm and thus the potential XPath syntax consequences are
sharply limited.

Are you guys very sure that you want to take this path? I can tell you now
that if we make this change and then there is some outcry after the next
ODK Collect release, SurveyCTO Collect will very likely stick with the new
change (since we will have changed pyxform and we'll then have too many
backward-compatibility concerns to then reverse the change).

Thanks,

Chris

··· On Mon, Nov 23, 2015 at 5:23 AM wrote:

Just for the history's sake, I found the time where the behavior changed.
It was at March 11, 2010:

https://bitbucket.org/m.sundt/javarosa/commits/b41875b2c23c9a21a868a5d2dc0ffc1f6b5b0b65#Lcore/src/org/javarosa/core/model/FormDef.javaT635

Take a look only in the changes of the "populateDynamicChoices" method.
The context reference used to be the "current" field, but after the changes
above it started be the field's parent (by adding getParent()). Perhaps
there was a reason for that change, but perhaps not, but I think all of us
agree that this should be fixed, even after 5 and a half years.

Meletis

On Monday, November 23, 2015 at 12:08:47 PM UTC+2, mel...@surveycto.com wrote:

Hi Mitch,

Please take a look at the code comment in line 31 here:

https://bitbucket.org/m.sundt/javarosa/src/bf17fb749b74e53915f086ca3827c3c6fe73fd07/core/src/org/javarosa/core/model/ItemsetBinding.java?at=default&fileviewer=file-view-default#ItemsetBinding.java-31

It clearly says: "ref of the control parent (group/formdef) of itemset
question
"

So, for some reason, they wanted it to point to the parent. I also see
that when the dynamic choices are populated, the evaluation context is
wrapped around that parent (contextRef object), that's why current() refers
to the parent group when evaluating itemsets nodesets.

I will run a few tests by changing the contextRef field value, I just
wanted to mention that in case you remember a reason for that code comment.

Meletis

On Friday, November 20, 2015 at 8:49:12 PM UTC+2, Mitch wrote:

Yes, I think it should be a one-line change; just don't know where.

Mitch

(I do jar-updates over Christmas / New-Years. Tedious, but doesn't
require a whole lot of attention.)

On Fri, Nov 20, 2015 at 10:36 AM, mel...@surveycto.com wrote:

Hi Mitch,

Thank you for all the details, I guess I was missing some of the
context of the original issue.

We will give it a shot to see if we can come up with a few ideas how to
fix the evaluation content of the itemset nodeset. I just hope it will not
be like last year with the "form processing logic" in JR, otherwise I
believe that every Christmas will be a JR Christmas. :slight_smile:

Meletis

On Friday, November 20, 2015 at 7:29:14 PM UTC+2, Mitch wrote:

The form was attached in one of my earlier replies.

On Fri, Nov 20, 2015 at 9:27 AM, Mitch Sundt mitche...@gmail.com wrote:

And you see this in constraint expressions where you write:

. > 6

I.e., current()/. should refer to the field, and not the group
containing the field, as it currently does when evaluating expressions in
the itemset nodeset.

On Fri, Nov 20, 2015 at 9:26 AM, Mitch Sundt mitche...@gmail.com wrote:

The issue is that there is a discrepancy in the meaning of '.'
across the different contexts of: calculate, relevant, , and these
itemset nodeset expressions.

In other usages, you need to reference this other field with
current()/../filterfield ; but in the nodeset expression, you would
reference it with current()/./filterfield

I.e., the notion of '.' is different when used within a nodeset
expression and when used within a constraint expression or an
field.

See the attached NBiggestOfSet form. For the label, it has:

    <select1 ref="/NBiggestOfSet/education/edu_level">
      <label>What is <output value="

current()/../referred_attendee_name"/>'s level of
education?
...

For a constraint, it has:

These each use the current()/../ to reference the group (
/NBiggesetOfSet/education ).

But, within the nodeset definition (if this form used one), you
would need to use current()/./referred_attendee_name to
reference a field in that group.

On Fri, Nov 20, 2015 at 8:22 AM, mel...@surveycto.com wrote:

Hi Mitch,

I think I'm missing something but I think that nothing is broken
and that the only issue is that the expression current()/../filterfield in
your example above should be just changed to current()/filterfield. If you
make that change, everything should work because it will be resolved to:

/data/group1/filterfield

Why should a "../" be present there? We have used
current()/somefield successfully in one of our test forms and it works
fine. And I think it would also work great in your example. Wouldn't it? Or
am I totally out of context?

Meletis

On Friday, November 20, 2015 at 5:35:16 PM UTC+2, Mitch wrote:

Hi Meletis,

That's not the issue --

The issue is that if you use a filter condition in the
nodeset specification of a field "myfield", within that nodeset
specification "current()/.." refers to the parent of the group enclosing
the "myfield" field. It does not refer to the group enclosing the
"myfield" field.

So if you have

a 4 fish b 14 fish ...

and if you have:

This will currently be broken. Because current()/../filterfield,
when evaluated within the itemset nodeset specification, is
resolved to:

/data/filterfield

instead of

/data/group1/filterfield

On Fri, Nov 20, 2015 at 1:58 AM, mel...@surveycto.com wrote:

Well, I think there is a fundamental issue here regarding
current() syntax. Please, let me remind you how JR works with "current()":

  1. The expression "current()" looks like a function but it is not
    treated like one; it is not handled by the XPathFuncExpr class, so, if you
    try to use it alone you will get an error that the "function" is not
    supported. In other words, you cannot use it alone in an expression in
    order to refer to the "current" field.

  2. When an expression includes "current()", then it ALWAYS needs
    to be used in an XPath, actually, the expression needs to BE an XPath
    expression, so that it can be handled with the XPathPathExpr class. For
    that reason the expression NEEDS to contain at least one "/".

  3. Taking 1 and 2 into consideration, the ONLY way to refer to
    the "current" field (named "somefield") is to use "current()/." or
    "current()/somefield". So, I looks like that the original implementation
    was "okay" in the sense that the "current()" should only be used in XPaths.

If we wanted to have "current()" refer to the current field, I
believe we should introduce a new function handler in XPathFuncExpr class.
Please let me know if this would work:

A. Whenever the "current()" expression is used inside an XPath,
then the current code would be used as it is used right now (maintaining
backwards compatibility), so that it will still be referring to the
enclosing group, not the field itself.
B. Whenever the "current()" expression is used inside a
non-XPath, then the new code would be used (in XPathFuncExpr) and that
would be evaluated in a way that the current field would be
returned/referenced, NOT the enclosing group.

That way, the following code could work, because it wouldn't
throw the "unrecognized function" error:

label .... ....

What do you think about that approach?

Meletis

On Friday, November 13, 2015 at 11:38:15 PM UTC+2, Martijn van de Rijdt wrote:

Hi,

I'd like to ask for a clarification about the current() node in
an XPath predicate. It seems JR considers this to be the
parent node of the current question. (Thanks to Chris for
sharing this.)

E.g. in this question below, current() equates to /data/group
and not to /data/group/options. I would have expected it to be the latter.

....
    <label>label</label>

    <itemset nodeset="instance('crop_list')/root/item[name =

current()*/../*crop]">

         ....

         ....

    </itemset>

Is this by design? Or is it a bug?

We ran into this by realizing that the seemingly more intuitive
predicate [name = current()*/../../*crop] is not working.

See this form
https://docs.google.com/spreadsheets/d/1A74yEHRxQF1zbIcQSrwvZydggUU7m5T3ZAnlR6_TLA4/edit#gid=0
that illustrates the behaviour nicely. If current() is used in a
calculation (even if it makes no sense to do so) this extra step down is
not taken, but in a predicate it is. So it seems the behaviour is not
related to the current() function but to how predicates are handled in JR.

Thanks,
Martijn

--

--
Mitch Sundt
Software Engineer
University of Washington
mitche...@gmail.com

--

--
Mitch Sundt
Software Engineer
University of Washington
mitche...@gmail.com

--
Mitch Sundt
Software Engineer
University of Washington
mitche...@gmail.com

--
Mitch Sundt
Software Engineer
University of Washington
mitche...@gmail.com

--
You received this message because you are subscribed to the Google
Groups "ODK Developers" group.
To unsubscribe from this group and stop receiving emails from it, send
an email to opendatakit-developers+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--
Mitch Sundt
Software Engineer
University of Washington
mitche...@gmail.com

--
You received this message because you are subscribed to the Google Groups
"ODK Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to opendatakit-developers+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

w.r.t. backward compatibility

I too am concerned.

A proposed solution:

  1. Add a command line argument or other mechanism to request that XLSForm
    inject a 'revision="2015-11-23"' attribute into the element. e.g.,

This is used to alter the behavior of the XForms engine to accommodate
backward-compatible bug behaviors.

We can't identify the bugs (e.g, 'legacy="itemset-parent"'), since that
would break older forms run on the new ODK Collect releases.
We have to advance the revision code as bugs are fixed.

I think an integer value is fine for this.

I don't anticipate making a lot of revisions, and I think the revisions
would be sequentially applied (you can't pick-and-choose what bugs you
include).

In the future, this would allow us to re-do the date-time behavior.

  1. Modify the XLSForm tools (e.g., the online form generator) to enable the
    user to select what revision they want to emit (much like we did last year
    with the JR form eval choice setting within ODK Collect). We can default
    to emitting the latest revision if the user does not make a choice.

  2. Augment JR to look for this revision attribute and store/restore it in
    the FormDef binary object.

Based upon this value, alter the behavior of the JR itemset evaluation to
fix the path bug.

  1. Modify ODK Validate to scan for syntax that may be impacted by a
    revision (to advise whether to change something). Not sure how complex
    that might be. May not be always possible?
··· ------------- *w.r.t. impacts of change*

Thank you Meletis for researching when this change occurred.

I will have to ponder what effect the parent/node change would have on
itemset copying. Hadn't thought that there was any relationship between
the two, but perhaps there is.

Yaw also reported that the Cascading Select itemsets on
opendatakit.appspot.com are not working with the 1.4.7 rev 1053 release
(and they work but give odd 'null' resolutions on 1057). This is in a
related area of this same code, so I will hopefully gain a better
understanding of dynamic itemsets as I uncover the causes of that problem.


Mitch

On Mon, Nov 23, 2015 at 5:59 AM, Christopher Robert crobert@surveycto.com wrote:

So just to be clear, the "fix" we're testing here reverts the entire
context within that populateDynamicChoices() function, to be the field
instead of the parent -- so the effects are potentially much wider than
just what current() returns. Potentially anything and everything that
depends on context can change now, since the context will shift from parent
to field.

This means that we revert a change from 5 years ago and alter the context
from now on. Neither Meletis nor I feel terribly confident that we can
entirely predict the consequences to backward compatibility. We're testing
the things we can think to test, but honestly it's XML users in the broader
ODK community who face the greatest risks here; SurveyCTO users by and
large use XLSForm and thus the potential XPath syntax consequences are
sharply limited.

Are you guys very sure that you want to take this path? I can tell you now
that if we make this change and then there is some outcry after the next
ODK Collect release, SurveyCTO Collect will very likely stick with the new
change (since we will have changed pyxform and we'll then have too many
backward-compatibility concerns to then reverse the change).

Thanks,

Chris

On Mon, Nov 23, 2015 at 5:23 AM meletis@surveycto.com wrote:

Just for the history's sake, I found the time where the behavior changed.
It was at March 11, 2010:

https://bitbucket.org/m.sundt/javarosa/commits/b41875b2c23c9a21a868a5d2dc0ffc1f6b5b0b65#Lcore/src/org/javarosa/core/model/FormDef.javaT635

Take a look only in the changes of the "populateDynamicChoices" method.
The context reference used to be the "current" field, but after the changes
above it started be the field's parent (by adding getParent()). Perhaps
there was a reason for that change, but perhaps not, but I think all of us
agree that this should be fixed, even after 5 and a half years.

Meletis

On Monday, November 23, 2015 at 12:08:47 PM UTC+2, mel...@surveycto.com wrote:

Hi Mitch,

Please take a look at the code comment in line 31 here:

https://bitbucket.org/m.sundt/javarosa/src/bf17fb749b74e53915f086ca3827c3c6fe73fd07/core/src/org/javarosa/core/model/ItemsetBinding.java?at=default&fileviewer=file-view-default#ItemsetBinding.java-31

It clearly says: "ref of the control parent (group/formdef) of itemset
question
"

So, for some reason, they wanted it to point to the parent. I also
see that when the dynamic choices are populated, the evaluation context is
wrapped around that parent (contextRef object), that's why current() refers
to the parent group when evaluating itemsets nodesets.

I will run a few tests by changing the contextRef field value, I just
wanted to mention that in case you remember a reason for that code comment.

Meletis

On Friday, November 20, 2015 at 8:49:12 PM UTC+2, Mitch wrote:

Yes, I think it should be a one-line change; just don't know where.

Mitch

(I do jar-updates over Christmas / New-Years. Tedious, but doesn't
require a whole lot of attention.)

On Fri, Nov 20, 2015 at 10:36 AM, mel...@surveycto.com wrote:

Hi Mitch,

Thank you for all the details, I guess I was missing some of the
context of the original issue.

We will give it a shot to see if we can come up with a few ideas how
to fix the evaluation content of the itemset nodeset. I just hope it will
not be like last year with the "form processing logic" in JR, otherwise I
believe that every Christmas will be a JR Christmas. :slight_smile:

Meletis

On Friday, November 20, 2015 at 7:29:14 PM UTC+2, Mitch wrote:

The form was attached in one of my earlier replies.

On Fri, Nov 20, 2015 at 9:27 AM, Mitch Sundt mitche...@gmail.com wrote:

And you see this in constraint expressions where you write:

. > 6

I.e., current()/. should refer to the field, and not the group
containing the field, as it currently does when evaluating expressions in
the itemset nodeset.

On Fri, Nov 20, 2015 at 9:26 AM, Mitch Sundt mitche...@gmail.com wrote:

The issue is that there is a discrepancy in the meaning of '.'
across the different contexts of: calculate, relevant, , and these
itemset nodeset expressions.

In other usages, you need to reference this other field with
current()/../filterfield ; but in the nodeset expression, you would
reference it with current()/./filterfield

I.e., the notion of '.' is different when used within a nodeset
expression and when used within a constraint expression or an
field.

See the attached NBiggestOfSet form. For the label, it has:

    <select1 ref="/NBiggestOfSet/education/edu_level">
      <label>What is <output value="

current()/../referred_attendee_name"/>'s level of
education?
...

For a constraint, it has:

These each use the current()/../ to reference the group (
/NBiggesetOfSet/education ).

But, within the nodeset definition (if this form used one), you
would need to use current()/./referred_attendee_name to
reference a field in that group.

On Fri, Nov 20, 2015 at 8:22 AM, mel...@surveycto.com wrote:

Hi Mitch,

I think I'm missing something but I think that nothing is broken
and that the only issue is that the expression current()/../filterfield in
your example above should be just changed to current()/filterfield. If you
make that change, everything should work because it will be resolved to:

/data/group1/filterfield

Why should a "../" be present there? We have used
current()/somefield successfully in one of our test forms and it works
fine. And I think it would also work great in your example. Wouldn't it? Or
am I totally out of context?

Meletis

On Friday, November 20, 2015 at 5:35:16 PM UTC+2, Mitch wrote:

Hi Meletis,

That's not the issue --

The issue is that if you use a filter condition in the
nodeset specification of a field "myfield", within that nodeset
specification "current()/.." refers to the parent of the group enclosing
the "myfield" field. It does not refer to the group enclosing the
"myfield" field.

So if you have

a 4 fish b 14 fish ...

and if you have:

This will currently be broken. Because current()/../filterfield,
when evaluated within the itemset nodeset specification, is
resolved to:

/data/filterfield

instead of

/data/group1/filterfield

On Fri, Nov 20, 2015 at 1:58 AM, mel...@surveycto.com wrote:

Well, I think there is a fundamental issue here regarding
current() syntax. Please, let me remind you how JR works with "current()":

  1. The expression "current()" looks like a function but it is
    not treated like one; it is not handled by the XPathFuncExpr class, so, if
    you try to use it alone you will get an error that the "function" is not
    supported. In other words, you cannot use it alone in an expression in
    order to refer to the "current" field.

  2. When an expression includes "current()", then it ALWAYS needs
    to be used in an XPath, actually, the expression needs to BE an XPath
    expression, so that it can be handled with the XPathPathExpr class. For
    that reason the expression NEEDS to contain at least one "/".

  3. Taking 1 and 2 into consideration, the ONLY way to refer to
    the "current" field (named "somefield") is to use "current()/." or
    "current()/somefield". So, I looks like that the original implementation
    was "okay" in the sense that the "current()" should only be used in XPaths.

If we wanted to have "current()" refer to the current field, I
believe we should introduce a new function handler in XPathFuncExpr class.
Please let me know if this would work:

A. Whenever the "current()" expression is used inside an XPath,
then the current code would be used as it is used right now (maintaining
backwards compatibility), so that it will still be referring to the
enclosing group, not the field itself.
B. Whenever the "current()" expression is used inside a
non-XPath, then the new code would be used (in XPathFuncExpr) and that
would be evaluated in a way that the current field would be
returned/referenced, NOT the enclosing group.

That way, the following code could work, because it wouldn't
throw the "unrecognized function" error:

label .... ....

What do you think about that approach?

Meletis

On Friday, November 13, 2015 at 11:38:15 PM UTC+2, Martijn van de Rijdt wrote:

Hi,

I'd like to ask for a clarification about the current() node in
an XPath predicate. It seems JR considers this to be the
parent node of the current question. (Thanks to Chris for
sharing this.)

E.g. in this question below, current() equates to /data/group
and not to /data/group/options. I would have expected it to be the latter.

....
    <label>label</label>

    <itemset nodeset="instance('crop_list')/root/item[name

= current()*/../*crop]">

         ....

         ....

    </itemset>

Is this by design? Or is it a bug?

We ran into this by realizing that the seemingly more intuitive
predicate [name = current()*/../../*crop] is not working.

See this form
https://docs.google.com/spreadsheets/d/1A74yEHRxQF1zbIcQSrwvZydggUU7m5T3ZAnlR6_TLA4/edit#gid=0
that illustrates the behaviour nicely. If current() is used in a
calculation (even if it makes no sense to do so) this extra step down is
not taken, but in a predicate it is. So it seems the behaviour is not
related to the current() function but to how predicates are handled in JR.

Thanks,
Martijn

--

--
Mitch Sundt
Software Engineer
University of Washington
mitche...@gmail.com

--

--
Mitch Sundt
Software Engineer
University of Washington
mitche...@gmail.com

--
Mitch Sundt
Software Engineer
University of Washington
mitche...@gmail.com

--
Mitch Sundt
Software Engineer
University of Washington
mitche...@gmail.com

--
You received this message because you are subscribed to the Google
Groups "ODK Developers" group.
To unsubscribe from this group and stop receiving emails from it, send
an email to opendatakit-developers+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--
Mitch Sundt
Software Engineer
University of Washington
mitche...@gmail.com

--
You received this message because you are subscribed to the Google Groups
"ODK Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to opendatakit-developers+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups
"ODK Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to opendatakit-developers+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--
Mitch Sundt
Software Engineer
University of Washington
mitchellsundt@gmail.com

And, if it isn't clear, to preserve backward-compatible operations, XLSForm
should not expand ${fieldname} with a relative path unless the revision
value is the newer one; if the revision is not specified, it would emit
absolute paths like it currently does.

I also switched to use a date value for the revision value ("2015-11-23")
which allows simple lexical comparisons.

··· On Mon, Nov 23, 2015 at 1:03 PM, Mitch Sundt wrote:

w.r.t. backward compatibility

I too am concerned.

A proposed solution:

  1. Add a command line argument or other mechanism to request that XLSForm
    inject a 'revision="2015-11-23"' attribute into the element. e.g.,

This is used to alter the behavior of the XForms engine to accommodate
backward-compatible bug behaviors.

We can't identify the bugs (e.g, 'legacy="itemset-parent"'), since that
would break older forms run on the new ODK Collect releases.
We have to advance the revision code as bugs are fixed.

I think an integer value is fine for this.

I don't anticipate making a lot of revisions, and I think the revisions
would be sequentially applied (you can't pick-and-choose what bugs you
include).

In the future, this would allow us to re-do the date-time behavior.

  1. Modify the XLSForm tools (e.g., the online form generator) to enable
    the user to select what revision they want to emit (much like we did last
    year with the JR form eval choice setting within ODK Collect). We can
    default to emitting the latest revision if the user does not make a choice.

  2. Augment JR to look for this revision attribute and store/restore it in
    the FormDef binary object.

Based upon this value, alter the behavior of the JR itemset evaluation to
fix the path bug.

  1. Modify ODK Validate to scan for syntax that may be impacted by a
    revision (to advise whether to change something). Not sure how complex
    that might be. May not be always possible?

w.r.t. impacts of change

Thank you Meletis for researching when this change occurred.

I will have to ponder what effect the parent/node change would have on
itemset copying. Hadn't thought that there was any relationship between
the two, but perhaps there is.

Yaw also reported that the Cascading Select itemsets on
opendatakit.appspot.com are not working with the 1.4.7 rev 1053 release
(and they work but give odd 'null' resolutions on 1057). This is in a
related area of this same code, so I will hopefully gain a better
understanding of dynamic itemsets as I uncover the causes of that problem.


Mitch

On Mon, Nov 23, 2015 at 5:59 AM, Christopher Robert <crobert@surveycto.com wrote:

So just to be clear, the "fix" we're testing here reverts the entire
context within that populateDynamicChoices() function, to be the field
instead of the parent -- so the effects are potentially much wider than
just what current() returns. Potentially anything and everything that
depends on context can change now, since the context will shift from parent
to field.

This means that we revert a change from 5 years ago and alter the context
from now on. Neither Meletis nor I feel terribly confident that we can
entirely predict the consequences to backward compatibility. We're testing
the things we can think to test, but honestly it's XML users in the broader
ODK community who face the greatest risks here; SurveyCTO users by and
large use XLSForm and thus the potential XPath syntax consequences are
sharply limited.

Are you guys very sure that you want to take this path? I can tell you
now that if we make this change and then there is some outcry after the
next ODK Collect release, SurveyCTO Collect will very likely stick with the
new change (since we will have changed pyxform and we'll then have too many
backward-compatibility concerns to then reverse the change).

Thanks,

Chris

On Mon, Nov 23, 2015 at 5:23 AM meletis@surveycto.com wrote:

Just for the history's sake, I found the time where the behavior
changed. It was at March 11, 2010:

https://bitbucket.org/m.sundt/javarosa/commits/b41875b2c23c9a21a868a5d2dc0ffc1f6b5b0b65#Lcore/src/org/javarosa/core/model/FormDef.javaT635

Take a look only in the changes of the "populateDynamicChoices" method.
The context reference used to be the "current" field, but after the changes
above it started be the field's parent (by adding getParent()). Perhaps
there was a reason for that change, but perhaps not, but I think all of us
agree that this should be fixed, even after 5 and a half years.

Meletis

On Monday, November 23, 2015 at 12:08:47 PM UTC+2, mel...@surveycto.com wrote:

Hi Mitch,

Please take a look at the code comment in line 31 here:

https://bitbucket.org/m.sundt/javarosa/src/bf17fb749b74e53915f086ca3827c3c6fe73fd07/core/src/org/javarosa/core/model/ItemsetBinding.java?at=default&fileviewer=file-view-default#ItemsetBinding.java-31

It clearly says: "ref of the control parent (group/formdef) of
itemset question
"

So, for some reason, they wanted it to point to the parent. I also
see that when the dynamic choices are populated, the evaluation context is
wrapped around that parent (contextRef object), that's why current() refers
to the parent group when evaluating itemsets nodesets.

I will run a few tests by changing the contextRef field value, I just
wanted to mention that in case you remember a reason for that code comment.

Meletis

On Friday, November 20, 2015 at 8:49:12 PM UTC+2, Mitch wrote:

Yes, I think it should be a one-line change; just don't know where.

Mitch

(I do jar-updates over Christmas / New-Years. Tedious, but doesn't
require a whole lot of attention.)

On Fri, Nov 20, 2015 at 10:36 AM, mel...@surveycto.com wrote:

Hi Mitch,

Thank you for all the details, I guess I was missing some of the
context of the original issue.

We will give it a shot to see if we can come up with a few ideas how
to fix the evaluation content of the itemset nodeset. I just hope it will
not be like last year with the "form processing logic" in JR, otherwise I
believe that every Christmas will be a JR Christmas. :slight_smile:

Meletis

On Friday, November 20, 2015 at 7:29:14 PM UTC+2, Mitch wrote:

The form was attached in one of my earlier replies.

On Fri, Nov 20, 2015 at 9:27 AM, Mitch Sundt mitche...@gmail.com wrote:

And you see this in constraint expressions where you write:

. > 6

I.e., current()/. should refer to the field, and not the group
containing the field, as it currently does when evaluating expressions in
the itemset nodeset.

On Fri, Nov 20, 2015 at 9:26 AM, Mitch Sundt mitche...@gmail.com wrote:

The issue is that there is a discrepancy in the meaning of '.'
across the different contexts of: calculate, relevant, , and these
itemset nodeset expressions.

In other usages, you need to reference this other field with
current()/../filterfield ; but in the nodeset expression, you would
reference it with current()/./filterfield

I.e., the notion of '.' is different when used within a nodeset
expression and when used within a constraint expression or an
field.

See the attached NBiggestOfSet form. For the label, it has:

    <select1 ref="/NBiggestOfSet/education/edu_level">
      <label>What is <output value="

current()/../referred_attendee_name"/>'s level of
education?
...

For a constraint, it has:

These each use the current()/../ to reference the group (
/NBiggesetOfSet/education ).

But, within the nodeset definition (if this form used one), you
would need to use current()/./referred_attendee_name to
reference a field in that group.

On Fri, Nov 20, 2015 at 8:22 AM, mel...@surveycto.com wrote:

Hi Mitch,

I think I'm missing something but I think that nothing is broken
and that the only issue is that the expression current()/../filterfield in
your example above should be just changed to current()/filterfield. If you
make that change, everything should work because it will be resolved to:

/data/group1/filterfield

Why should a "../" be present there? We have used
current()/somefield successfully in one of our test forms and it works
fine. And I think it would also work great in your example. Wouldn't it? Or
am I totally out of context?

Meletis

On Friday, November 20, 2015 at 5:35:16 PM UTC+2, Mitch wrote:

Hi Meletis,

That's not the issue --

The issue is that if you use a filter condition in the
nodeset specification of a field "myfield", within that nodeset
specification "current()/.." refers to the parent of the group enclosing
the "myfield" field. It does not refer to the group enclosing the
"myfield" field.

So if you have

a 4 fish b 14 fish ...

and if you have:

This will currently be broken. Because
current()/../filterfield,
when evaluated within the itemset nodeset specification, is
resolved to:

/data/filterfield

instead of

/data/group1/filterfield

On Fri, Nov 20, 2015 at 1:58 AM, mel...@surveycto.com wrote:

Well, I think there is a fundamental issue here regarding
current() syntax. Please, let me remind you how JR works with "current()":

  1. The expression "current()" looks like a function but it is
    not treated like one; it is not handled by the XPathFuncExpr class, so, if
    you try to use it alone you will get an error that the "function" is not
    supported. In other words, you cannot use it alone in an expression in
    order to refer to the "current" field.

  2. When an expression includes "current()", then it ALWAYS
    needs to be used in an XPath, actually, the expression needs to BE an XPath
    expression, so that it can be handled with the XPathPathExpr class. For
    that reason the expression NEEDS to contain at least one "/".

  3. Taking 1 and 2 into consideration, the ONLY way to refer to
    the "current" field (named "somefield") is to use "current()/." or
    "current()/somefield". So, I looks like that the original implementation
    was "okay" in the sense that the "current()" should only be used in XPaths.

If we wanted to have "current()" refer to the current field, I
believe we should introduce a new function handler in XPathFuncExpr class.
Please let me know if this would work:

A. Whenever the "current()" expression is used inside an XPath,
then the current code would be used as it is used right now (maintaining
backwards compatibility), so that it will still be referring to the
enclosing group, not the field itself.
B. Whenever the "current()" expression is used inside a
non-XPath, then the new code would be used (in XPathFuncExpr) and that
would be evaluated in a way that the current field would be
returned/referenced, NOT the enclosing group.

That way, the following code could work, because it wouldn't
throw the "unrecognized function" error:

label .... ....

What do you think about that approach?

Meletis

On Friday, November 13, 2015 at 11:38:15 PM UTC+2, Martijn van de Rijdt wrote:

Hi,

I'd like to ask for a clarification about the current() node
in an XPath predicate. It seems JR considers this to be the
parent node of the current question. (Thanks to Chris for
sharing this.)

E.g. in this question below, current() equates to /data/group
and not to /data/group/options. I would have expected it to be the latter.

....
    <label>label</label>

    <itemset nodeset="instance('crop_list')/root/item[name

= current()*/../*crop]">

         ....

         ....

    </itemset>

Is this by design? Or is it a bug?

We ran into this by realizing that the seemingly more
intuitive predicate [name = current()*/../../*crop] is not
working.

See this form
https://docs.google.com/spreadsheets/d/1A74yEHRxQF1zbIcQSrwvZydggUU7m5T3ZAnlR6_TLA4/edit#gid=0
that illustrates the behaviour nicely. If current() is used in a
calculation (even if it makes no sense to do so) this extra step down is
not taken, but in a predicate it is. So it seems the behaviour is not
related to the current() function but to how predicates are handled in JR.

Thanks,
Martijn

--

--
Mitch Sundt
Software Engineer
University of Washington
mitche...@gmail.com

--

--
Mitch Sundt
Software Engineer
University of Washington
mitche...@gmail.com

--
Mitch Sundt
Software Engineer
University of Washington
mitche...@gmail.com

--
Mitch Sundt
Software Engineer
University of Washington
mitche...@gmail.com

--
You received this message because you are subscribed to the Google
Groups "ODK Developers" group.
To unsubscribe from this group and stop receiving emails from it,
send an email to opendatakit-developers+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--
Mitch Sundt
Software Engineer
University of Washington
mitche...@gmail.com

--
You received this message because you are subscribed to the Google
Groups "ODK Developers" group.
To unsubscribe from this group and stop receiving emails from it, send
an email to opendatakit-developers+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups
"ODK Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to opendatakit-developers+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--
Mitch Sundt
Software Engineer
University of Washington
mitchellsundt@gmail.com

--
Mitch Sundt
Software Engineer
University of Washington
mitchellsundt@gmail.com

Okay, well, this is turning into a bigger and bigger project and we may
well miss the window for our 2.01 maintenance release (and/or make the
changes so extensive and scary that we cannot in good conscience include
them in a maintenance release).

However, your idea for making JR sensitive to the revision value emitted by
pyxform opens a new possibility: we could release the pyxform change with
2.01 so that users can begin to benefit from the relative paths (relying on
today's current() behavior), then assume that when JR is "fixed" it will
only be fixed for XML emitted by newer pyxforms that would have updated
their paths to account for the fix (using your revision-value idea). The
primary drawback to this approach is that we would basically break all
cascading-select filter expressions for SurveyCTO+Enketo users (since
Martijn clearly wants to implement the "fixed" version of current() rather
than the JavaRosa-compatible version of current()).

Meletis also found another issue relating to the evaluation context in
jr:choice-name() (https://github.com/opendatakit/opendatakit/issues/1180),
so we have that to contend with as well. If we can't safely resolve the
context for jr:choice-name(), then that would present another major
drawback and further lessen our enthusiasm for this entire adventure.

Best,

Chris

··· On Mon, Nov 23, 2015 at 5:45 PM Mitch Sundt wrote:

And, if it isn't clear, to preserve backward-compatible operations,
XLSForm should not expand ${fieldname} with a relative path unless the
revision value is the newer one; if the revision is not specified, it would
emit absolute paths like it currently does.

I also switched to use a date value for the revision value ("2015-11-23")
which allows simple lexical comparisons.

On Mon, Nov 23, 2015 at 1:03 PM, Mitch Sundt mitchellsundt@gmail.com wrote:

w.r.t. backward compatibility

I too am concerned.

A proposed solution:

  1. Add a command line argument or other mechanism to request that XLSForm
    inject a 'revision="2015-11-23"' attribute into the element. e.g.,

This is used to alter the behavior of the XForms engine to accommodate
backward-compatible bug behaviors.

We can't identify the bugs (e.g, 'legacy="itemset-parent"'), since that
would break older forms run on the new ODK Collect releases.
We have to advance the revision code as bugs are fixed.

I think an integer value is fine for this.

I don't anticipate making a lot of revisions, and I think the revisions
would be sequentially applied (you can't pick-and-choose what bugs you
include).

In the future, this would allow us to re-do the date-time behavior.

  1. Modify the XLSForm tools (e.g., the online form generator) to enable
    the user to select what revision they want to emit (much like we did last
    year with the JR form eval choice setting within ODK Collect). We can
    default to emitting the latest revision if the user does not make a choice.

  2. Augment JR to look for this revision attribute and store/restore it in
    the FormDef binary object.

Based upon this value, alter the behavior of the JR itemset evaluation to
fix the path bug.

  1. Modify ODK Validate to scan for syntax that may be impacted by a
    revision (to advise whether to change something). Not sure how complex
    that might be. May not be always possible?

w.r.t. impacts of change

Thank you Meletis for researching when this change occurred.

I will have to ponder what effect the parent/node change would have on
itemset copying. Hadn't thought that there was any relationship between
the two, but perhaps there is.

Yaw also reported that the Cascading Select itemsets on
opendatakit.appspot.com are not working with the 1.4.7 rev 1053 release
(and they work but give odd 'null' resolutions on 1057). This is in a
related area of this same code, so I will hopefully gain a better
understanding of dynamic itemsets as I uncover the causes of that problem.


Mitch

On Mon, Nov 23, 2015 at 5:59 AM, Christopher Robert < crobert@surveycto.com> wrote:

So just to be clear, the "fix" we're testing here reverts the entire
context within that populateDynamicChoices() function, to be the field
instead of the parent -- so the effects are potentially much wider than
just what current() returns. Potentially anything and everything that
depends on context can change now, since the context will shift from parent
to field.

This means that we revert a change from 5 years ago and alter the
context from now on. Neither Meletis nor I feel terribly confident that we
can entirely predict the consequences to backward compatibility. We're
testing the things we can think to test, but honestly it's XML users in the
broader ODK community who face the greatest risks here; SurveyCTO users by
and large use XLSForm and thus the potential XPath syntax consequences are
sharply limited.

Are you guys very sure that you want to take this path? I can tell you
now that if we make this change and then there is some outcry after the
next ODK Collect release, SurveyCTO Collect will very likely stick with the
new change (since we will have changed pyxform and we'll then have too many
backward-compatibility concerns to then reverse the change).

Thanks,

Chris

On Mon, Nov 23, 2015 at 5:23 AM meletis@surveycto.com wrote:

Just for the history's sake, I found the time where the behavior
changed. It was at March 11, 2010:

https://bitbucket.org/m.sundt/javarosa/commits/b41875b2c23c9a21a868a5d2dc0ffc1f6b5b0b65#Lcore/src/org/javarosa/core/model/FormDef.javaT635

Take a look only in the changes of the "populateDynamicChoices" method.
The context reference used to be the "current" field, but after the changes
above it started be the field's parent (by adding getParent()). Perhaps
there was a reason for that change, but perhaps not, but I think all of us
agree that this should be fixed, even after 5 and a half years.

Meletis

On Monday, November 23, 2015 at 12:08:47 PM UTC+2, mel...@surveycto.com wrote:

Hi Mitch,

Please take a look at the code comment in line 31 here:

https://bitbucket.org/m.sundt/javarosa/src/bf17fb749b74e53915f086ca3827c3c6fe73fd07/core/src/org/javarosa/core/model/ItemsetBinding.java?at=default&fileviewer=file-view-default#ItemsetBinding.java-31

It clearly says: "ref of the control parent (group/formdef) of
itemset question
"

So, for some reason, they wanted it to point to the parent. I also
see that when the dynamic choices are populated, the evaluation context is
wrapped around that parent (contextRef object), that's why current() refers
to the parent group when evaluating itemsets nodesets.

I will run a few tests by changing the contextRef field value, I just
wanted to mention that in case you remember a reason for that code comment.

Meletis

On Friday, November 20, 2015 at 8:49:12 PM UTC+2, Mitch wrote:

Yes, I think it should be a one-line change; just don't know where.

Mitch

(I do jar-updates over Christmas / New-Years. Tedious, but doesn't
require a whole lot of attention.)

On Fri, Nov 20, 2015 at 10:36 AM, mel...@surveycto.com wrote:

Hi Mitch,

Thank you for all the details, I guess I was missing some of the
context of the original issue.

We will give it a shot to see if we can come up with a few ideas how
to fix the evaluation content of the itemset nodeset. I just hope it will
not be like last year with the "form processing logic" in JR, otherwise I
believe that every Christmas will be a JR Christmas. :slight_smile:

Meletis

On Friday, November 20, 2015 at 7:29:14 PM UTC+2, Mitch wrote:

The form was attached in one of my earlier replies.

On Fri, Nov 20, 2015 at 9:27 AM, Mitch Sundt mitche...@gmail.com wrote:

And you see this in constraint expressions where you write:

. > 6

I.e., current()/. should refer to the field, and not the group
containing the field, as it currently does when evaluating expressions in
the itemset nodeset.

On Fri, Nov 20, 2015 at 9:26 AM, Mitch Sundt mitche...@gmail.com wrote:

The issue is that there is a discrepancy in the meaning of '.'
across the different contexts of: calculate, relevant, , and these
itemset nodeset expressions.

In other usages, you need to reference this other field with
current()/../filterfield ; but in the nodeset expression, you would
reference it with current()/./filterfield

I.e., the notion of '.' is different when used within a nodeset
expression and when used within a constraint expression or an
field.

See the attached NBiggestOfSet form. For the label, it has:

    <select1 ref="/NBiggestOfSet/education/edu_level">
      <label>What is <output value="

current()/../referred_attendee_name"/>'s level of
education?
...

For a constraint, it has:

These each use the current()/../ to reference the group (
/NBiggesetOfSet/education ).

But, within the nodeset definition (if this form used one), you
would need to use current()/./referred_attendee_name to
reference a field in that group.

On Fri, Nov 20, 2015 at 8:22 AM, mel...@surveycto.com wrote:

Hi Mitch,

I think I'm missing something but I think that nothing is broken
and that the only issue is that the expression current()/../filterfield in
your example above should be just changed to current()/filterfield. If you
make that change, everything should work because it will be resolved to:

/data/group1/filterfield

Why should a "../" be present there? We have used
current()/somefield successfully in one of our test forms and it works
fine. And I think it would also work great in your example. Wouldn't it? Or
am I totally out of context?

Meletis

On Friday, November 20, 2015 at 5:35:16 PM UTC+2, Mitch wrote:

Hi Meletis,

That's not the issue --

The issue is that if you use a filter condition in the
nodeset specification of a field "myfield", within that nodeset
specification "current()/.." refers to the parent of the group enclosing
the "myfield" field. It does not refer to the group enclosing the
"myfield" field.

So if you have

a 4 fish b 14 fish ...

and if you have:

This will currently be broken. Because
current()/../filterfield,
when evaluated within the itemset nodeset specification, is
resolved to:

/data/filterfield

instead of

/data/group1/filterfield

On Fri, Nov 20, 2015 at 1:58 AM, mel...@surveycto.com wrote:

Well, I think there is a fundamental issue here regarding
current() syntax. Please, let me remind you how JR works with "current()":

  1. The expression "current()" looks like a function but it is
    not treated like one; it is not handled by the XPathFuncExpr class, so, if
    you try to use it alone you will get an error that the "function" is not
    supported. In other words, you cannot use it alone in an expression in
    order to refer to the "current" field.

  2. When an expression includes "current()", then it ALWAYS
    needs to be used in an XPath, actually, the expression needs to BE an XPath
    expression, so that it can be handled with the XPathPathExpr class. For
    that reason the expression NEEDS to contain at least one "/".

  3. Taking 1 and 2 into consideration, the ONLY way to refer to
    the "current" field (named "somefield") is to use "current()/." or
    "current()/somefield". So, I looks like that the original implementation
    was "okay" in the sense that the "current()" should only be used in XPaths.

If we wanted to have "current()" refer to the current field, I
believe we should introduce a new function handler in XPathFuncExpr class.
Please let me know if this would work:

A. Whenever the "current()" expression is used inside an
XPath, then the current code would be used as it is used right now
(maintaining backwards compatibility), so that it will still be referring
to the enclosing group, not the field itself.
B. Whenever the "current()" expression is used inside a
non-XPath, then the new code would be used (in XPathFuncExpr) and that
would be evaluated in a way that the current field would be
returned/referenced, NOT the enclosing group.

That way, the following code could work, because it wouldn't
throw the "unrecognized function" error:

label .... ....

What do you think about that approach?

Meletis

On Friday, November 13, 2015 at 11:38:15 PM UTC+2, Martijn van de Rijdt wrote:

Hi,

I'd like to ask for a clarification about the current() node
in an XPath predicate. It seems JR considers this to be the
parent node of the current question. (Thanks to Chris for
sharing this.)

E.g. in this question below, current() equates to /data/group
and not to /data/group/options. I would have expected it to be the latter.

....
    <label>label</label>

    <itemset nodeset="instance('crop_list')/root/item[name

= current()*/../*crop]">

         ....

         ....

    </itemset>

Is this by design? Or is it a bug?

We ran into this by realizing that the seemingly more
intuitive predicate [name = current()*/../../*crop] is not
working.

See this form
https://docs.google.com/spreadsheets/d/1A74yEHRxQF1zbIcQSrwvZydggUU7m5T3ZAnlR6_TLA4/edit#gid=0
that illustrates the behaviour nicely. If current() is used in a
calculation (even if it makes no sense to do so) this extra step down is
not taken, but in a predicate it is. So it seems the behaviour is not
related to the current() function but to how predicates are handled in JR.

Thanks,
Martijn

--

--
Mitch Sundt
Software Engineer
University of Washington
mitche...@gmail.com

--

--
Mitch Sundt
Software Engineer
University of Washington
mitche...@gmail.com

--
Mitch Sundt
Software Engineer
University of Washington
mitche...@gmail.com

--
Mitch Sundt
Software Engineer
University of Washington
mitche...@gmail.com

--
You received this message because you are subscribed to the Google
Groups "ODK Developers" group.
To unsubscribe from this group and stop receiving emails from it,
send an email to opendatakit-developers+unsubscribe@googlegroups.com
.
For more options, visit https://groups.google.com/d/optout.

--
Mitch Sundt
Software Engineer
University of Washington
mitche...@gmail.com

--
You received this message because you are subscribed to the Google
Groups "ODK Developers" group.
To unsubscribe from this group and stop receiving emails from it, send
an email to opendatakit-developers+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google
Groups "ODK Developers" group.
To unsubscribe from this group and stop receiving emails from it, send
an email to opendatakit-developers+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--
Mitch Sundt
Software Engineer
University of Washington
mitchellsundt@gmail.com

--
Mitch Sundt
Software Engineer
University of Washington
mitchellsundt@gmail.com

--
You received this message because you are subscribed to the Google Groups
"ODK Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to opendatakit-developers+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

I appreciate the creative solution. It does sound like it will become a bit
of a headache to manage and explain to users though. Whether it's worth it
depends on how many active forms are out there that rely on the bug. We
should get Ona's and KoBo's input on this too, I think.

From Enketo's side, relative paths were never supported in choice-filters
(to my embarrassment and surprise - which led to my question). As a result,
there is no significant issue with backwards compatibility in Enketo once
it is fixed in JR. Not sure how meaningful this is to ODK Collect, but no
Enketo user has ever reported this lack of support. I think to date it's
been too hard to craft such choice-filters. Because of the way Enketo is
integrated into KoBo, Ona, Formhub, I'd be surprised if there were a
significant amount of those users using relative choice-filters.

I do indeed feel strongly about following the XForms 2.0 spec here. I think
relative path support is going to be very useful in the future, and I agree
that we need to look at the bigger picture and do this correctly, so it
won't catch up with us.

··· On Tue, Nov 24, 2015 at 4:24 AM, Christopher Robert wrote:

Okay, well, this is turning into a bigger and bigger project and we may
well miss the window for our 2.01 maintenance release (and/or make the
changes so extensive and scary that we cannot in good conscience include
them in a maintenance release).

However, your idea for making JR sensitive to the revision value emitted
by pyxform opens a new possibility: we could release the pyxform change
with 2.01 so that users can begin to benefit from the relative paths
(relying on today's current() behavior), then assume that when JR is
"fixed" it will only be fixed for XML emitted by newer pyxforms that would
have updated their paths to account for the fix (using your revision-value
idea). The primary drawback to this approach is that we would basically
break all cascading-select filter expressions for SurveyCTO+Enketo users
(since Martijn clearly wants to implement the "fixed" version of current()
rather than the JavaRosa-compatible version of current()).

Meletis also found another issue relating to the evaluation context in
jr:choice-name() (https://github.com/opendatakit/opendatakit/issues/1180),
so we have that to contend with as well. If we can't safely resolve the
context for jr:choice-name(), then that would present another major
drawback and further lessen our enthusiasm for this entire adventure.

Best,

Chris

On Mon, Nov 23, 2015 at 5:45 PM Mitch Sundt mitchellsundt@gmail.com wrote:

And, if it isn't clear, to preserve backward-compatible operations,
XLSForm should not expand ${fieldname} with a relative path unless the
revision value is the newer one; if the revision is not specified, it would
emit absolute paths like it currently does.

I also switched to use a date value for the revision value ("2015-11-23")
which allows simple lexical comparisons.

On Mon, Nov 23, 2015 at 1:03 PM, Mitch Sundt mitchellsundt@gmail.com wrote:

w.r.t. backward compatibility

I too am concerned.

A proposed solution:

  1. Add a command line argument or other mechanism to request that
    XLSForm inject a 'revision="2015-11-23"' attribute into the
    element. e.g.,

This is used to alter the behavior of the XForms engine to accommodate
backward-compatible bug behaviors.

We can't identify the bugs (e.g, 'legacy="itemset-parent"'), since that
would break older forms run on the new ODK Collect releases.
We have to advance the revision code as bugs are fixed.

I think an integer value is fine for this.

I don't anticipate making a lot of revisions, and I think the revisions
would be sequentially applied (you can't pick-and-choose what bugs you
include).

In the future, this would allow us to re-do the date-time behavior.

  1. Modify the XLSForm tools (e.g., the online form generator) to enable
    the user to select what revision they want to emit (much like we did last
    year with the JR form eval choice setting within ODK Collect). We can
    default to emitting the latest revision if the user does not make a choice.

  2. Augment JR to look for this revision attribute and store/restore it
    in the FormDef binary object.

Based upon this value, alter the behavior of the JR itemset evaluation
to fix the path bug.

  1. Modify ODK Validate to scan for syntax that may be impacted by a
    revision (to advise whether to change something). Not sure how complex
    that might be. May not be always possible?

w.r.t. impacts of change

Thank you Meletis for researching when this change occurred.

I will have to ponder what effect the parent/node change would have on
itemset copying. Hadn't thought that there was any relationship between
the two, but perhaps there is.

Yaw also reported that the Cascading Select itemsets on
opendatakit.appspot.com are not working with the 1.4.7 rev 1053 release
(and they work but give odd 'null' resolutions on 1057). This is in a
related area of this same code, so I will hopefully gain a better
understanding of dynamic itemsets as I uncover the causes of that problem.


Mitch

On Mon, Nov 23, 2015 at 5:59 AM, Christopher Robert < crobert@surveycto.com> wrote:

So just to be clear, the "fix" we're testing here reverts the entire
context within that populateDynamicChoices() function, to be the field
instead of the parent -- so the effects are potentially much wider than
just what current() returns. Potentially anything and everything that
depends on context can change now, since the context will shift from parent
to field.

This means that we revert a change from 5 years ago and alter the
context from now on. Neither Meletis nor I feel terribly confident that we
can entirely predict the consequences to backward compatibility. We're
testing the things we can think to test, but honestly it's XML users in the
broader ODK community who face the greatest risks here; SurveyCTO users by
and large use XLSForm and thus the potential XPath syntax consequences are
sharply limited.

Are you guys very sure that you want to take this path? I can tell you
now that if we make this change and then there is some outcry after the
next ODK Collect release, SurveyCTO Collect will very likely stick with the
new change (since we will have changed pyxform and we'll then have too many
backward-compatibility concerns to then reverse the change).

Thanks,

Chris

On Mon, Nov 23, 2015 at 5:23 AM meletis@surveycto.com wrote:

Just for the history's sake, I found the time where the behavior
changed. It was at March 11, 2010:

https://bitbucket.org/m.sundt/javarosa/commits/b41875b2c23c9a21a868a5d2dc0ffc1f6b5b0b65#Lcore/src/org/javarosa/core/model/FormDef.javaT635

Take a look only in the changes of the "populateDynamicChoices"
method. The context reference used to be the "current" field, but after the
changes above it started be the field's parent (by adding getParent()).
Perhaps there was a reason for that change, but perhaps not, but I think
all of us agree that this should be fixed, even after 5 and a half years.

Meletis

On Monday, November 23, 2015 at 12:08:47 PM UTC+2, mel...@surveycto.com wrote:

Hi Mitch,

Please take a look at the code comment in line 31 here:

https://bitbucket.org/m.sundt/javarosa/src/bf17fb749b74e53915f086ca3827c3c6fe73fd07/core/src/org/javarosa/core/model/ItemsetBinding.java?at=default&fileviewer=file-view-default#ItemsetBinding.java-31

It clearly says: "ref of the control parent (group/formdef) of
itemset question
"

So, for some reason, they wanted it to point to the parent. I also
see that when the dynamic choices are populated, the evaluation context is
wrapped around that parent (contextRef object), that's why current() refers
to the parent group when evaluating itemsets nodesets.

I will run a few tests by changing the contextRef field value, I just
wanted to mention that in case you remember a reason for that code comment.

Meletis

On Friday, November 20, 2015 at 8:49:12 PM UTC+2, Mitch wrote:

Yes, I think it should be a one-line change; just don't know where.

Mitch

(I do jar-updates over Christmas / New-Years. Tedious, but doesn't
require a whole lot of attention.)

On Fri, Nov 20, 2015 at 10:36 AM, mel...@surveycto.com wrote:

Hi Mitch,

Thank you for all the details, I guess I was missing some of the
context of the original issue.

We will give it a shot to see if we can come up with a few ideas
how to fix the evaluation content of the itemset nodeset. I just hope it
will not be like last year with the "form processing logic" in JR,
otherwise I believe that every Christmas will be a JR Christmas. :slight_smile:

Meletis

On Friday, November 20, 2015 at 7:29:14 PM UTC+2, Mitch wrote:

The form was attached in one of my earlier replies.

On Fri, Nov 20, 2015 at 9:27 AM, Mitch Sundt mitche...@gmail.com wrote:

And you see this in constraint expressions where you write:

. > 6

I.e., current()/. should refer to the field, and not the group
containing the field, as it currently does when evaluating expressions in
the itemset nodeset.

On Fri, Nov 20, 2015 at 9:26 AM, Mitch Sundt <mitche...@gmail.com wrote:

The issue is that there is a discrepancy in the meaning of
'.' across the different contexts of: calculate, relevant, , and
these itemset nodeset expressions.

In other usages, you need to reference this other field with
current()/../filterfield ; but in the nodeset expression, you would
reference it with current()/./filterfield

I.e., the notion of '.' is different when used within a nodeset
expression and when used within a constraint expression or an
field.

See the attached NBiggestOfSet form. For the label, it has:

    <select1 ref="/NBiggestOfSet/education/edu_level">
      <label>What is <output value="

current()/../referred_attendee_name"/>'s level of
education?
...

For a constraint, it has:

These each use the current()/../ to reference the group (
/NBiggesetOfSet/education ).

But, within the nodeset definition (if this form used one), you
would need to use current()/./referred_attendee_name to
reference a field in that group.

On Fri, Nov 20, 2015 at 8:22 AM, mel...@surveycto.com wrote:

Hi Mitch,

I think I'm missing something but I think that nothing is
broken and that the only issue is that the
expression current()/../filterfield in your example above should be just
changed to current()/filterfield. If you make that change, everything
should work because it will be resolved to:

/data/group1/filterfield

Why should a "../" be present there? We have used
current()/somefield successfully in one of our test forms and it works
fine. And I think it would also work great in your example. Wouldn't it? Or
am I totally out of context?

Meletis

On Friday, November 20, 2015 at 5:35:16 PM UTC+2, Mitch wrote:

Hi Meletis,

That's not the issue --

The issue is that if you use a filter condition in the
nodeset specification of a field "myfield", within that nodeset
specification "current()/.." refers to the parent of the group enclosing
the "myfield" field. It does not refer to the group enclosing the
"myfield" field.

So if you have

a 4 fish b 14 fish ...

and if you have:

This will currently be broken. Because
current()/../filterfield,
when evaluated within the itemset nodeset specification, is
resolved to:

/data/filterfield

instead of

/data/group1/filterfield

On Fri, Nov 20, 2015 at 1:58 AM, mel...@surveycto.com wrote:

Well, I think there is a fundamental issue here regarding
current() syntax. Please, let me remind you how JR works with "current()":

  1. The expression "current()" looks like a function but it is
    not treated like one; it is not handled by the XPathFuncExpr class, so, if
    you try to use it alone you will get an error that the "function" is not
    supported. In other words, you cannot use it alone in an expression in
    order to refer to the "current" field.

  2. When an expression includes "current()", then it ALWAYS
    needs to be used in an XPath, actually, the expression needs to BE an XPath
    expression, so that it can be handled with the XPathPathExpr class. For
    that reason the expression NEEDS to contain at least one "/".

  3. Taking 1 and 2 into consideration, the ONLY way to refer
    to the "current" field (named "somefield") is to use "current()/." or
    "current()/somefield". So, I looks like that the original implementation
    was "okay" in the sense that the "current()" should only be used in XPaths.

If we wanted to have "current()" refer to the current field,
I believe we should introduce a new function handler in XPathFuncExpr
class. Please let me know if this would work:

A. Whenever the "current()" expression is used inside an
XPath, then the current code would be used as it is used right now
(maintaining backwards compatibility), so that it will still be referring
to the enclosing group, not the field itself.
B. Whenever the "current()" expression is used inside a
non-XPath, then the new code would be used (in XPathFuncExpr) and that
would be evaluated in a way that the current field would be
returned/referenced, NOT the enclosing group.

That way, the following code could work, because it wouldn't
throw the "unrecognized function" error:

label .... ....

What do you think about that approach?

Meletis

On Friday, November 13, 2015 at 11:38:15 PM UTC+2, Martijn van de Rijdt wrote:

Hi,

I'd like to ask for a clarification about the current() node
in an XPath predicate. It seems JR considers this to be the
parent node of the current question. (Thanks to Chris for
sharing this.)

E.g. in this question below, current() equates to
/data/group and not to /data/group/options. I would have expected it to be
the latter.

....
    <label>label</label>

    <itemset nodeset="instance('crop_list')/root/item[name

= current()*/../*crop]">

         ....

         ....

    </itemset>

Is this by design? Or is it a bug?

We ran into this by realizing that the seemingly more
intuitive predicate [name = current()*/../../*crop] is not
working.

See this form
https://docs.google.com/spreadsheets/d/1A74yEHRxQF1zbIcQSrwvZydggUU7m5T3ZAnlR6_TLA4/edit#gid=0
that illustrates the behaviour nicely. If current() is used in a
calculation (even if it makes no sense to do so) this extra step down is
not taken, but in a predicate it is. So it seems the behaviour is not
related to the current() function but to how predicates are handled in JR.

Thanks,
Martijn

--

--
Mitch Sundt
Software Engineer
University of Washington
mitche...@gmail.com

--

--
Mitch Sundt
Software Engineer
University of Washington
mitche...@gmail.com

--
Mitch Sundt
Software Engineer
University of Washington
mitche...@gmail.com

--
Mitch Sundt
Software Engineer
University of Washington
mitche...@gmail.com

--
You received this message because you are subscribed to the Google
Groups "ODK Developers" group.
To unsubscribe from this group and stop receiving emails from it,
send an email to
opendatakit-developers+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--
Mitch Sundt
Software Engineer
University of Washington
mitche...@gmail.com

--
You received this message because you are subscribed to the Google
Groups "ODK Developers" group.
To unsubscribe from this group and stop receiving emails from it, send
an email to opendatakit-developers+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google
Groups "ODK Developers" group.
To unsubscribe from this group and stop receiving emails from it, send
an email to opendatakit-developers+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--
Mitch Sundt
Software Engineer
University of Washington
mitchellsundt@gmail.com

--
Mitch Sundt
Software Engineer
University of Washington
mitchellsundt@gmail.com

--
You received this message because you are subscribed to the Google Groups
"ODK Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to opendatakit-developers+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to a topic in the
Google Groups "ODK Developers" group.
To unsubscribe from this topic, visit
https://groups.google.com/d/topic/opendatakit-developers/nN8HTA5pUIM/unsubscribe
.
To unsubscribe from this group and all its topics, send an email to
opendatakit-developers+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--

Revolutionizing data collection since 2012.

Enketo https://enketo.org/ | LinkedIn
http://www.linkedin.com/company/enketo-llc | GitHub
https://github.com/enketo | Twitter https://twitter.com/enketo
| Blog http://blog.enketo.org/

I agree. Having just delved into the itemset code, there are a number of
issues that would need to be cleaned up.

What I would propose is to leave JR / ODK Collect broken as-is. This will
be future work.

The python code will then be the focus of the tweaks.

(1) change:

json_survey = xls2json.workbook_to_json(json.loads(request.POST['workbookJson']),
form_name=form_name, warnings=warningsList)

to add an optional revision argument:

json_survey = xls2json.workbook_to_json(json.loads(request.POST['workbookJson']),
form_name=form_name, warnings=warningsList*, revision='2015-11-24')*

(2) change that implementation to:

(2a) store the revision string somewhere in the returned json object.

(2b) expand ${fieldname} with the new relative path expansion only if the
passed-in revision is >= '2015-11-24' ; otherwise, use the absolute path
expansion.

(3) change:

create_survey_element_from_dict()

to emit the revision value as an attribute on the element.

··· =============== The JR itemset bug will be fixed at a later date.

When that is fixed, we can introduce a new revision string for that
behavior change.

i.e., once we have the revision string moving through the system, we can
begin to support non-backward-compatible changes to behaviors in XLSForm
and JR form processing.

For the time being, the ODK XLSForm converter will not supply a revision
argument, and not make use of this pyxform enhancement. Once the JR fix is
done, we will then support that new revision and the current behavior,
defaulting to the new behavior, but supporting users in generating the old
behavior.

Mitch

On Tue, Nov 24, 2015 at 8:44 AM, Martijn van de Rijdt martijn@enketo.org wrote:

I appreciate the creative solution. It does sound like it will become a
bit of a headache to manage and explain to users though. Whether it's worth
it depends on how many active forms are out there that rely on the bug. We
should get Ona's and KoBo's input on this too, I think.

From Enketo's side, relative paths were never supported in choice-filters
(to my embarrassment and surprise - which led to my question). As a result,
there is no significant issue with backwards compatibility in Enketo once
it is fixed in JR. Not sure how meaningful this is to ODK Collect, but no
Enketo user has ever reported this lack of support. I think to date it's
been too hard to craft such choice-filters. Because of the way Enketo is
integrated into KoBo, Ona, Formhub, I'd be surprised if there were a
significant amount of those users using relative choice-filters.

I do indeed feel strongly about following the XForms 2.0 spec here. I
think relative path support is going to be very useful in the future, and I
agree that we need to look at the bigger picture and do this correctly, so
it won't catch up with us.

On Tue, Nov 24, 2015 at 4:24 AM, Christopher Robert <crobert@surveycto.com wrote:

Okay, well, this is turning into a bigger and bigger project and we may
well miss the window for our 2.01 maintenance release (and/or make the
changes so extensive and scary that we cannot in good conscience include
them in a maintenance release).

However, your idea for making JR sensitive to the revision value emitted
by pyxform opens a new possibility: we could release the pyxform change
with 2.01 so that users can begin to benefit from the relative paths
(relying on today's current() behavior), then assume that when JR is
"fixed" it will only be fixed for XML emitted by newer pyxforms that would
have updated their paths to account for the fix (using your revision-value
idea). The primary drawback to this approach is that we would basically
break all cascading-select filter expressions for SurveyCTO+Enketo users
(since Martijn clearly wants to implement the "fixed" version of current()
rather than the JavaRosa-compatible version of current()).

Meletis also found another issue relating to the evaluation context in
jr:choice-name() (https://github.com/opendatakit/opendatakit/issues/1180),
so we have that to contend with as well. If we can't safely resolve the
context for jr:choice-name(), then that would present another major
drawback and further lessen our enthusiasm for this entire adventure.

Best,

Chris

On Mon, Nov 23, 2015 at 5:45 PM Mitch Sundt mitchellsundt@gmail.com wrote:

And, if it isn't clear, to preserve backward-compatible operations,
XLSForm should not expand ${fieldname} with a relative path unless the
revision value is the newer one; if the revision is not specified, it would
emit absolute paths like it currently does.

I also switched to use a date value for the revision value
("2015-11-23") which allows simple lexical comparisons.

On Mon, Nov 23, 2015 at 1:03 PM, Mitch Sundt mitchellsundt@gmail.com wrote:

w.r.t. backward compatibility

I too am concerned.

A proposed solution:

  1. Add a command line argument or other mechanism to request that
    XLSForm inject a 'revision="2015-11-23"' attribute into the
    element. e.g.,

This is used to alter the behavior of the XForms engine to accommodate
backward-compatible bug behaviors.

We can't identify the bugs (e.g, 'legacy="itemset-parent"'), since that
would break older forms run on the new ODK Collect releases.
We have to advance the revision code as bugs are fixed.

I think an integer value is fine for this.

I don't anticipate making a lot of revisions, and I think the revisions
would be sequentially applied (you can't pick-and-choose what bugs you
include).

In the future, this would allow us to re-do the date-time behavior.

  1. Modify the XLSForm tools (e.g., the online form generator) to enable
    the user to select what revision they want to emit (much like we did last
    year with the JR form eval choice setting within ODK Collect). We can
    default to emitting the latest revision if the user does not make a choice.

  2. Augment JR to look for this revision attribute and store/restore it
    in the FormDef binary object.

Based upon this value, alter the behavior of the JR itemset evaluation
to fix the path bug.

  1. Modify ODK Validate to scan for syntax that may be impacted by a
    revision (to advise whether to change something). Not sure how complex
    that might be. May not be always possible?

w.r.t. impacts of change

Thank you Meletis for researching when this change occurred.

I will have to ponder what effect the parent/node change would have on
itemset copying. Hadn't thought that there was any relationship between
the two, but perhaps there is.

Yaw also reported that the Cascading Select itemsets on
opendatakit.appspot.com are not working with the 1.4.7 rev 1053
release (and they work but give odd 'null' resolutions on 1057). This is in
a related area of this same code, so I will hopefully gain a better
understanding of dynamic itemsets as I uncover the causes of that problem.


Mitch

On Mon, Nov 23, 2015 at 5:59 AM, Christopher Robert < crobert@surveycto.com> wrote:

So just to be clear, the "fix" we're testing here reverts the entire
context within that populateDynamicChoices() function, to be the field
instead of the parent -- so the effects are potentially much wider than
just what current() returns. Potentially anything and everything that
depends on context can change now, since the context will shift from parent
to field.

This means that we revert a change from 5 years ago and alter the
context from now on. Neither Meletis nor I feel terribly confident that we
can entirely predict the consequences to backward compatibility. We're
testing the things we can think to test, but honestly it's XML users in the
broader ODK community who face the greatest risks here; SurveyCTO users by
and large use XLSForm and thus the potential XPath syntax consequences are
sharply limited.

Are you guys very sure that you want to take this path? I can tell you
now that if we make this change and then there is some outcry after the
next ODK Collect release, SurveyCTO Collect will very likely stick with the
new change (since we will have changed pyxform and we'll then have too many
backward-compatibility concerns to then reverse the change).

Thanks,

Chris

On Mon, Nov 23, 2015 at 5:23 AM meletis@surveycto.com wrote:

Just for the history's sake, I found the time where the behavior
changed. It was at March 11, 2010:

https://bitbucket.org/m.sundt/javarosa/commits/b41875b2c23c9a21a868a5d2dc0ffc1f6b5b0b65#Lcore/src/org/javarosa/core/model/FormDef.javaT635

Take a look only in the changes of the "populateDynamicChoices"
method. The context reference used to be the "current" field, but after the
changes above it started be the field's parent (by adding getParent()).
Perhaps there was a reason for that change, but perhaps not, but I think
all of us agree that this should be fixed, even after 5 and a half years.

Meletis

On Monday, November 23, 2015 at 12:08:47 PM UTC+2, mel...@surveycto.com wrote:

Hi Mitch,

Please take a look at the code comment in line 31 here:

https://bitbucket.org/m.sundt/javarosa/src/bf17fb749b74e53915f086ca3827c3c6fe73fd07/core/src/org/javarosa/core/model/ItemsetBinding.java?at=default&fileviewer=file-view-default#ItemsetBinding.java-31

It clearly says: "ref of the control parent (group/formdef) of
itemset question
"

So, for some reason, they wanted it to point to the parent. I
also see that when the dynamic choices are populated, the evaluation
context is wrapped around that parent (contextRef object), that's why
current() refers to the parent group when evaluating itemsets nodesets.

I will run a few tests by changing the contextRef field value, I
just wanted to mention that in case you remember a reason for that code
comment.

Meletis

On Friday, November 20, 2015 at 8:49:12 PM UTC+2, Mitch wrote:

Yes, I think it should be a one-line change; just don't know where.

Mitch

(I do jar-updates over Christmas / New-Years. Tedious, but doesn't
require a whole lot of attention.)

On Fri, Nov 20, 2015 at 10:36 AM, mel...@surveycto.com wrote:

Hi Mitch,

Thank you for all the details, I guess I was missing some of the
context of the original issue.

We will give it a shot to see if we can come up with a few ideas
how to fix the evaluation content of the itemset nodeset. I just hope it
will not be like last year with the "form processing logic" in JR,
otherwise I believe that every Christmas will be a JR Christmas. :slight_smile:

Meletis

On Friday, November 20, 2015 at 7:29:14 PM UTC+2, Mitch wrote:

The form was attached in one of my earlier replies.

On Fri, Nov 20, 2015 at 9:27 AM, Mitch Sundt <mitche...@gmail.com wrote:

And you see this in constraint expressions where you write:

. > 6

I.e., current()/. should refer to the field, and not the group
containing the field, as it currently does when evaluating expressions in
the itemset nodeset.

On Fri, Nov 20, 2015 at 9:26 AM, Mitch Sundt < mitche...@gmail.com> wrote:

The issue is that there is a discrepancy in the meaning of
'.' across the different contexts of: calculate, relevant, , and
these itemset nodeset expressions.

In other usages, you need to reference this other field with
current()/../filterfield ; but in the nodeset expression, you would
reference it with current()/./filterfield

I.e., the notion of '.' is different when used within a nodeset
expression and when used within a constraint expression or an
field.

See the attached NBiggestOfSet form. For the label, it has:

    <select1 ref="/NBiggestOfSet/education/edu_level">
      <label>What is <output value="

current()/../referred_attendee_name"/>'s level of
education?
...

For a constraint, it has:

These each use the current()/../ to reference the group (
/NBiggesetOfSet/education ).

But, within the nodeset definition (if this form used one), you
would need to use current()/./referred_attendee_name to
reference a field in that group.

On Fri, Nov 20, 2015 at 8:22 AM, mel...@surveycto.com wrote:

Hi Mitch,

I think I'm missing something but I think that nothing is
broken and that the only issue is that the
expression current()/../filterfield in your example above should be just
changed to current()/filterfield. If you make that change, everything
should work because it will be resolved to:

/data/group1/filterfield

Why should a "../" be present there? We have used
current()/somefield successfully in one of our test forms and it works
fine. And I think it would also work great in your example. Wouldn't it? Or
am I totally out of context?

Meletis

On Friday, November 20, 2015 at 5:35:16 PM UTC+2, Mitch wrote:

Hi Meletis,

That's not the issue --

The issue is that if you use a filter condition in the
nodeset specification of a field "myfield", within that nodeset
specification "current()/.." refers to the parent of the group enclosing
the "myfield" field. It does not refer to the group enclosing the
"myfield" field.

So if you have

a 4 fish b 14 fish ...

and if you have:

This will currently be broken. Because
current()/../filterfield,
when evaluated within the itemset nodeset specification, is
resolved to:

/data/filterfield

instead of

/data/group1/filterfield

On Fri, Nov 20, 2015 at 1:58 AM, mel...@surveycto.com wrote:

Well, I think there is a fundamental issue here regarding
current() syntax. Please, let me remind you how JR works with "current()":

  1. The expression "current()" looks like a function but it
    is not treated like one; it is not handled by the XPathFuncExpr class, so,
    if you try to use it alone you will get an error that the "function" is not
    supported. In other words, you cannot use it alone in an expression in
    order to refer to the "current" field.

  2. When an expression includes "current()", then it ALWAYS
    needs to be used in an XPath, actually, the expression needs to BE an XPath
    expression, so that it can be handled with the XPathPathExpr class. For
    that reason the expression NEEDS to contain at least one "/".

  3. Taking 1 and 2 into consideration, the ONLY way to refer
    to the "current" field (named "somefield") is to use "current()/." or
    "current()/somefield". So, I looks like that the original implementation
    was "okay" in the sense that the "current()" should only be used in XPaths.

If we wanted to have "current()" refer to the current field,
I believe we should introduce a new function handler in XPathFuncExpr
class. Please let me know if this would work:

A. Whenever the "current()" expression is used inside an
XPath, then the current code would be used as it is used right now
(maintaining backwards compatibility), so that it will still be referring
to the enclosing group, not the field itself.
B. Whenever the "current()" expression is used inside a
non-XPath, then the new code would be used (in XPathFuncExpr) and that
would be evaluated in a way that the current field would be
returned/referenced, NOT the enclosing group.

That way, the following code could work, because it wouldn't
throw the "unrecognized function" error:

label .... ....

What do you think about that approach?

Meletis

On Friday, November 13, 2015 at 11:38:15 PM UTC+2, Martijn van de Rijdt wrote:

Hi,

I'd like to ask for a clarification about the current()
node in an XPath predicate. It seems JR considers this to be the
parent node of the current question. (Thanks to Chris
for sharing this.)

E.g. in this question below, current() equates to
/data/group and not to /data/group/options. I would have expected it to be
the latter.

....
    <label>label</label>

    <itemset nodeset="instance('crop_list')/root/item[name

= current()*/../*crop]">

         ....

         ....

    </itemset>

Is this by design? Or is it a bug?

We ran into this by realizing that the seemingly more
intuitive predicate [name = current()*/../../*crop] is not
working.

See this form
https://docs.google.com/spreadsheets/d/1A74yEHRxQF1zbIcQSrwvZydggUU7m5T3ZAnlR6_TLA4/edit#gid=0
that illustrates the behaviour nicely. If current() is used in a
calculation (even if it makes no sense to do so) this extra step down is
not taken, but in a predicate it is. So it seems the behaviour is not
related to the current() function but to how predicates are handled in JR.

Thanks,
Martijn

--

--
Mitch Sundt
Software Engineer
University of Washington
mitche...@gmail.com

--

--
Mitch Sundt
Software Engineer
University of Washington
mitche...@gmail.com

--
Mitch Sundt
Software Engineer
University of Washington
mitche...@gmail.com

--
Mitch Sundt
Software Engineer
University of Washington
mitche...@gmail.com

--
You received this message because you are subscribed to the Google
Groups "ODK Developers" group.
To unsubscribe from this group and stop receiving emails from it,
send an email to
opendatakit-developers+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--
Mitch Sundt
Software Engineer
University of Washington
mitche...@gmail.com

--
You received this message because you are subscribed to the Google
Groups "ODK Developers" group.
To unsubscribe from this group and stop receiving emails from it,
send an email to opendatakit-developers+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google
Groups "ODK Developers" group.
To unsubscribe from this group and stop receiving emails from it, send
an email to opendatakit-developers+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--
Mitch Sundt
Software Engineer
University of Washington
mitchellsundt@gmail.com

--
Mitch Sundt
Software Engineer
University of Washington
mitchellsundt@gmail.com

--
You received this message because you are subscribed to the Google
Groups "ODK Developers" group.
To unsubscribe from this group and stop receiving emails from it, send
an email to opendatakit-developers+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to a topic in the
Google Groups "ODK Developers" group.
To unsubscribe from this topic, visit
https://groups.google.com/d/topic/opendatakit-developers/nN8HTA5pUIM/unsubscribe
.
To unsubscribe from this group and all its topics, send an email to
opendatakit-developers+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--
Revolutionizing data collection since 2012.

Enketo https://enketo.org/ | LinkedIn
http://www.linkedin.com/company/enketo-llc | GitHub
https://github.com/enketo | Twitter https://twitter.com/enketo
| Blog http://blog.enketo.org/

--
You received this message because you are subscribed to the Google Groups
"ODK Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to opendatakit-developers+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--
Mitch Sundt
Software Engineer
University of Washington
mitchellsundt@gmail.com

Hi, Just catching up on the discussion. Sorry if I'm a bit behind in my
understanding of things.

Regarding the pyxform changes--

  1. Instead of the date, could we have the configuration parameter describe
    the specific change?

So instead of:

revision="2015-11-24"

we specify something like:

current_references_parent=True (default to False with the next pyxform
version bump)

  1. Could we put this configuration into the xlsform's settings sheet? I
    think this would be easier to test and configure. We could also add a way
    to override the setting with an extra argument to the CLI or xls2json call.
··· On Tuesday, November 24, 2015 at 3:26:18 PM UTC-5, Mitch wrote: > > I agree. Having just delved into the itemset code, there are a number of > issues that would need to be cleaned up. > > What I would propose is to leave JR / ODK Collect broken as-is. This will > be future work. > > The python code will then be the focus of the tweaks. > > (1) change: > > json_survey = *xls2json.workbook_to_json*(json.loads(request.POST['workbookJson']), > form_name=form_name, warnings=warningsList) > > to add an optional revision argument: > > json_survey = *xls2json.workbook_to_json*(json.loads(request.POST['workbookJson']), > form_name=form_name, warnings=warningsList*, revision='2015-11-24')* > > > (2) change that implementation to: > > (2a) store the revision string somewhere in the returned json object. > > (2b) expand ${fieldname} with the new relative path expansion only if the > passed-in revision is >= '2015-11-24' ; otherwise, use the absolute path > expansion. > > (3) change: > > create_survey_element_from_dict() > > to emit the revision value as an attribute on the element. > > =============== > The JR itemset bug will be fixed at a later date. > > When that is fixed, we can introduce a new revision string for that > behavior change. > > i.e., once we have the revision string moving through the system, we can > begin to support non-backward-compatible changes to behaviors in XLSForm > and JR form processing. > > For the time being, the ODK XLSForm converter will not supply a revision > argument, and not make use of this pyxform enhancement. Once the JR fix is > done, we will then support that new revision and the current behavior, > defaulting to the new behavior, but supporting users in generating the old > behavior. > > Mitch > > > On Tue, Nov 24, 2015 at 8:44 AM, Martijn van de Rijdt <mar...@enketo.org > wrote: > >> I appreciate the creative solution. It does sound like it will become a >> bit of a headache to manage and explain to users though. Whether it's worth >> it depends on how many active forms are out there that rely on the bug. We >> should get Ona's and KoBo's input on this too, I think. >> >> From Enketo's side, relative paths were never supported in choice-filters >> (to my embarrassment and surprise - which led to my question). As a result, >> there is no significant issue with backwards compatibility in Enketo once >> it is fixed in JR. Not sure how meaningful this is to ODK Collect, but no >> Enketo user has ever reported this lack of support. I think to date it's >> been too hard to craft such choice-filters. Because of the way Enketo is >> integrated into KoBo, Ona, Formhub, I'd be surprised if there were a >> significant amount of those users using relative choice-filters. >> >> I do indeed feel strongly about following the XForms 2.0 spec here. I >> think relative path support is going to be very useful in the future, and I >> agree that we need to look at the bigger picture and do this correctly, so >> it won't catch up with us. >> >> On Tue, Nov 24, 2015 at 4:24 AM, Christopher Robert <cro...@surveycto.com > wrote: >> >>> Okay, well, this is turning into a bigger and bigger project and we may >>> well miss the window for our 2.01 maintenance release (and/or make the >>> changes so extensive and scary that we cannot in good conscience include >>> them in a maintenance release). >>> >>> However, your idea for making JR sensitive to the revision value emitted >>> by pyxform opens a new possibility: we could release the pyxform change >>> with 2.01 so that users can begin to benefit from the relative paths >>> (relying on today's current() behavior), then assume that when JR is >>> "fixed" it will only be fixed for XML emitted by newer pyxforms that would >>> have updated their paths to account for the fix (using your revision-value >>> idea). The primary drawback to this approach is that we would basically >>> break all cascading-select filter expressions for SurveyCTO+Enketo users >>> (since Martijn clearly wants to implement the "fixed" version of current() >>> rather than the JavaRosa-compatible version of current()). >>> >>> Meletis also found another issue relating to the evaluation context in >>> jr:choice-name() (https://github.com/opendatakit/opendatakit/issues/1180), >>> so we have that to contend with as well. If we can't safely resolve the >>> context for jr:choice-name(), then that would present another major >>> drawback and further lessen our enthusiasm for this entire adventure. >>> >>> Best, >>> >>> Chris >>> >>> On Mon, Nov 23, 2015 at 5:45 PM Mitch Sundt <mitche...@gmail.com > wrote: >>> >>>> And, if it isn't clear, to preserve backward-compatible operations, >>>> XLSForm should not expand ${fieldname} with a relative path unless the >>>> revision value is the newer one; if the revision is not specified, it would >>>> emit absolute paths like it currently does. >>>> >>>> I also switched to use a date value for the revision value >>>> ("2015-11-23") which allows simple lexical comparisons. >>>> >>>> >>>> On Mon, Nov 23, 2015 at 1:03 PM, Mitch Sundt <mitche...@gmail.com > wrote: >>>> >>>>> *w.r.t. backward compatibility* >>>>> >>>>> I too am concerned. >>>>> >>>>> A proposed solution: >>>>> >>>>> 1. Add a command line argument or other mechanism to request that >>>>> XLSForm inject a 'revision="2015-11-23"' attribute into the >>>>> element. e.g., >>>>> >>>>> This is used to alter the behavior of the XForms engine to accommodate >>>>> backward-compatible bug behaviors. >>>>> >>>>> We can't identify the bugs (e.g, 'legacy="itemset-parent"'), since >>>>> that would break older forms run on the new ODK Collect releases. >>>>> We have to advance the revision code as bugs are fixed. >>>>> >>>>> I think an integer value is fine for this. >>>>> >>>>> I don't anticipate making a lot of revisions, and I think the >>>>> revisions would be sequentially applied (you can't pick-and-choose what >>>>> bugs you include). >>>>> >>>>> In the future, this would allow us to re-do the date-time behavior. >>>>> >>>>> 2. Modify the XLSForm tools (e.g., the online form generator) to >>>>> enable the user to select what revision they want to emit (much like we did >>>>> last year with the JR form eval choice setting within ODK Collect). We can >>>>> default to emitting the latest revision if the user does not make a choice. >>>>> >>>>> 3. Augment JR to look for this revision attribute and store/restore it >>>>> in the FormDef binary object. >>>>> >>>>> Based upon this value, alter the behavior of the JR itemset evaluation >>>>> to fix the path bug. >>>>> >>>>> 4. Modify ODK Validate to scan for syntax that may be impacted by a >>>>> revision (to advise whether to change something). Not sure how complex >>>>> that might be. May not be always possible? >>>>> >>>>> ------------- >>>>> *w.r.t. impacts of change* >>>>> >>>>> Thank you Meletis for researching when this change occurred. >>>>> >>>>> I will have to ponder what effect the parent/node change would have on >>>>> itemset copying. Hadn't thought that there was any relationship between >>>>> the two, but perhaps there is. >>>>> >>>>> Yaw also reported that the Cascading Select itemsets on >>>>> opendatakit.appspot.com are not working with the 1.4.7 rev 1053 >>>>> release (and they work but give odd 'null' resolutions on 1057). This is in >>>>> a related area of this same code, so I will hopefully gain a better >>>>> understanding of dynamic itemsets as I uncover the causes of that problem. >>>>> >>>>> ------------- >>>>> Mitch >>>>> >>>>> On Mon, Nov 23, 2015 at 5:59 AM, Christopher Robert < cro...@surveycto.com > wrote: >>>>> >>>>>> So just to be clear, the "fix" we're testing here reverts the entire >>>>>> context within that populateDynamicChoices() function, to be the field >>>>>> instead of the parent -- so the effects are potentially much wider than >>>>>> just what current() returns. Potentially anything and everything that >>>>>> depends on context can change now, since the context will shift from parent >>>>>> to field. >>>>>> >>>>>> This means that we revert a change from 5 years ago and alter the >>>>>> context from now on. Neither Meletis nor I feel terribly confident that we >>>>>> can entirely predict the consequences to backward compatibility. We're >>>>>> testing the things we can think to test, but honestly it's XML users in the >>>>>> broader ODK community who face the greatest risks here; SurveyCTO users by >>>>>> and large use XLSForm and thus the potential XPath syntax consequences are >>>>>> sharply limited. >>>>>> >>>>>> Are you guys very sure that you want to take this path? I can tell >>>>>> you now that if we make this change and then there is some outcry after the >>>>>> next ODK Collect release, SurveyCTO Collect will very likely stick with the >>>>>> new change (since we will have changed pyxform and we'll then have too many >>>>>> backward-compatibility concerns to then reverse the change). >>>>>> >>>>>> Thanks, >>>>>> >>>>>> Chris >>>>>> >>>>>> >>>>>> On Mon, Nov 23, 2015 at 5:23 AM <mel...@surveycto.com > wrote: >>>>>> >>>>>>> Just for the history's sake, I found the time where the behavior >>>>>>> changed. It was at March 11, 2010: >>>>>>> >>>>>>> >>>>>>> https://bitbucket.org/m.sundt/javarosa/commits/b41875b2c23c9a21a868a5d2dc0ffc1f6b5b0b65#Lcore/src/org/javarosa/core/model/FormDef.javaT635 >>>>>>> >>>>>>> Take a look only in the changes of the "populateDynamicChoices" >>>>>>> method. The context reference used to be the "current" field, but after the >>>>>>> changes above it started be the field's parent (by adding getParent()). >>>>>>> Perhaps there was a reason for that change, but perhaps not, but I think >>>>>>> all of us agree that this should be fixed, even after 5 and a half years. >>>>>>> >>>>>>> Meletis >>>>>>> >>>>>>> On Monday, November 23, 2015 at 12:08:47 PM UTC+2, mel...@surveycto.com wrote: >>>>>>>> >>>>>>>> Hi Mitch, >>>>>>>> >>>>>>>> Please take a look at the code comment in line 31 here: >>>>>>>> >>>>>>>> >>>>>>>> https://bitbucket.org/m.sundt/javarosa/src/bf17fb749b74e53915f086ca3827c3c6fe73fd07/core/src/org/javarosa/core/model/ItemsetBinding.java?at=default&fileviewer=file-view-default#ItemsetBinding.java-31 >>>>>>>> >>>>>>>> It clearly says: "*ref of the control parent (group/formdef) of >>>>>>>> itemset question*" >>>>>>>> >>>>>>>> So, for some reason, they *wanted* it to point to the parent. I >>>>>>>> also see that when the dynamic choices are populated, the evaluation >>>>>>>> context is wrapped around that parent (contextRef object), that's why >>>>>>>> current() refers to the parent group when evaluating itemsets nodesets. >>>>>>>> >>>>>>>> I will run a few tests by changing the contextRef field value, I >>>>>>>> just wanted to mention that in case you remember a reason for that code >>>>>>>> comment. >>>>>>>> >>>>>>>> Meletis >>>>>>>> >>>>>>>> On Friday, November 20, 2015 at 8:49:12 PM UTC+2, Mitch wrote: >>>>>>>>> >>>>>>>>> Yes, I think it should be a one-line change; just don't know where. >>>>>>>>> >>>>>>>>> Mitch >>>>>>>>> >>>>>>>>> (I do jar-updates over Christmas / New-Years. Tedious, but >>>>>>>>> doesn't require a whole lot of attention.) >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> On Fri, Nov 20, 2015 at 10:36 AM, wrote: >>>>>>>>> >>>>>>>>>> Hi Mitch, >>>>>>>>>> >>>>>>>>>> Thank you for all the details, I guess I was missing some of the >>>>>>>>>> context of the original issue. >>>>>>>>>> >>>>>>>>>> We will give it a shot to see if we can come up with a few ideas >>>>>>>>>> how to fix the evaluation content of the itemset nodeset. I just hope it >>>>>>>>>> will not be like last year with the "form processing logic" in JR, >>>>>>>>>> otherwise I believe that every Christmas will be a JR Christmas. :) >>>>>>>>>> >>>>>>>>>> Meletis >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> On Friday, November 20, 2015 at 7:29:14 PM UTC+2, Mitch wrote: >>>>>>>>>>> >>>>>>>>>>> The form was attached in one of my earlier replies. >>>>>>>>>>> >>>>>>>>>>> On Fri, Nov 20, 2015 at 9:27 AM, Mitch Sundt < mitche...@gmail.com> wrote: >>>>>>>>>>> >>>>>>>>>>>> And you see this in constraint expressions where you write: >>>>>>>>>>>> >>>>>>>>>>>> . > 6 >>>>>>>>>>>> >>>>>>>>>>>> I.e., current()/. should refer to the field, and not the group >>>>>>>>>>>> containing the field, as it currently does when evaluating expressions in >>>>>>>>>>>> the itemset nodeset. >>>>>>>>>>>> >>>>>>>>>>>> On Fri, Nov 20, 2015 at 9:26 AM, Mitch Sundt < mitche...@gmail.com> wrote: >>>>>>>>>>>> >>>>>>>>>>>>> The issue is that there is a *discrepancy* in the meaning of >>>>>>>>>>>>> '.' across the different contexts of: calculate, relevant, , and >>>>>>>>>>>>> these itemset nodeset expressions. >>>>>>>>>>>>> >>>>>>>>>>>>> In other usages, you need to reference this other field with >>>>>>>>>>>>> current()/../filterfield ; but in the nodeset expression, you would >>>>>>>>>>>>> reference it with current()/./filterfield >>>>>>>>>>>>> >>>>>>>>>>>>> I.e., the notion of '.' is different when used within a >>>>>>>>>>>>> nodeset expression and when used within a constraint expression or an >>>>>>>>>>>>> field. >>>>>>>>>>>>> >>>>>>>>>>>>> See the attached NBiggestOfSet form. For the label, it has: >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> What is 's level of >>>>>>>>>>>>> education? >>>>>>>>>>>>> ... >>>>>>>>>>>>> >>>>>>>>>>>>> For a constraint, it has: >>>>>>>>>>>>> >>>>>>>>>>>> nodeset="/NBiggestOfSet/education/referred_attendee_name" readonly="true()" >>>>>>>>>>>>> type="string"/> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> These each use the *current()/../* to reference the group ( >>>>>>>>>>>>> /NBiggesetOfSet/education ). >>>>>>>>>>>>> >>>>>>>>>>>>> But, within the nodeset definition (if this form used one), >>>>>>>>>>>>> you would need to use *current()/./referred_attendee_name* >>>>>>>>>>>>> to reference a field in that group. >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> On Fri, Nov 20, 2015 at 8:22 AM, wrote: >>>>>>>>>>>>> >>>>>>>>>>>>>> Hi Mitch, >>>>>>>>>>>>>> >>>>>>>>>>>>>> I think I'm missing something but I think that nothing is >>>>>>>>>>>>>> broken and that the only issue is that the >>>>>>>>>>>>>> expression current()/../filterfield in your example above should be just >>>>>>>>>>>>>> changed to current()/filterfield. If you make that change, everything >>>>>>>>>>>>>> should work because it will be resolved to: >>>>>>>>>>>>>> >>>>>>>>>>>>>> /data/group1/filterfield >>>>>>>>>>>>>> >>>>>>>>>>>>>> Why should a "../" be present there? We have used >>>>>>>>>>>>>> current()/somefield successfully in one of our test forms and it works >>>>>>>>>>>>>> fine. And I think it would also work great in your example. Wouldn't it? Or >>>>>>>>>>>>>> am I totally out of context? >>>>>>>>>>>>>> >>>>>>>>>>>>>> Meletis >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> On Friday, November 20, 2015 at 5:35:16 PM UTC+2, Mitch wrote: >>>>>>>>>>>>>> >>>>>>>>>>>>>>> Hi Meletis, >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> That's not the issue -- >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> The issue is that if you use a filter condition in the >>>>>>>>>>>>>>> nodeset specification of a field "myfield", within that nodeset >>>>>>>>>>>>>>> specification "current()/.." refers to the parent of the group enclosing >>>>>>>>>>>>>>> the "myfield" field. It does not refer to the group enclosing the >>>>>>>>>>>>>>> "myfield" field. >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> So if you have >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> a >>>>>>>>>>>>>>> 4 >>>>>>>>>>>>>>> fish >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> b >>>>>>>>>>>>>>> 14 >>>>>>>>>>>>>>> fish >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> ... >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> and if you have: >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>> nodeset="instance('food')/root/fooditem[filtercriteria=current()/../filterfield]"> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> This will currently be broken. Because >>>>>>>>>>>>>>> current()/../filterfield, >>>>>>>>>>>>>>> when evaluated within the itemset nodeset specification, is >>>>>>>>>>>>>>> resolved to: >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> /data/filterfield >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> instead of >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> /data/group1/filterfield >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> On Fri, Nov 20, 2015 at 1:58 AM, wrote: >>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> Well, I think there is a fundamental issue here regarding >>>>>>>>>>>>>>>> current() syntax. Please, let me remind you how JR works with "current()": >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> 1. The expression "current()" looks like a function but it >>>>>>>>>>>>>>>> is not treated like one; it is not handled by the XPathFuncExpr class, so, >>>>>>>>>>>>>>>> if you try to use it alone you will get an error that the "function" is not >>>>>>>>>>>>>>>> supported. In other words, you cannot use it alone in an expression in >>>>>>>>>>>>>>>> order to refer to the "current" field. >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> 2. When an expression includes "current()", then it ALWAYS >>>>>>>>>>>>>>>> needs to be used in an XPath, actually, the expression needs to BE an XPath >>>>>>>>>>>>>>>> expression, so that it can be handled with the XPathPathExpr class. For >>>>>>>>>>>>>>>> that reason the expression NEEDS to contain at least one "/". >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> 3. Taking 1 and 2 into consideration, the ONLY way to refer >>>>>>>>>>>>>>>> to the "current" field (named "somefield") is to use "current()/." or >>>>>>>>>>>>>>>> "current()/somefield". So, I looks like that the original implementation >>>>>>>>>>>>>>>> was "okay" in the sense that the "current()" should only be used in XPaths. >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> If we wanted to have "current()" refer to the current >>>>>>>>>>>>>>>> field, I believe we should introduce a new function handler in >>>>>>>>>>>>>>>> XPathFuncExpr class. Please let me know if this would work: >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> A. Whenever the "current()" expression is used inside an >>>>>>>>>>>>>>>> XPath, then the current code would be used as it is used right now >>>>>>>>>>>>>>>> (maintaining backwards compatibility), so that it will still be referring >>>>>>>>>>>>>>>> to the enclosing group, not the field itself. >>>>>>>>>>>>>>>> B. Whenever the "current()" expression is used inside a >>>>>>>>>>>>>>>> non-XPath, then the new code would be used (in XPathFuncExpr) and that >>>>>>>>>>>>>>>> would be evaluated in a way that the current field would be >>>>>>>>>>>>>>>> returned/referenced, NOT the enclosing group. >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> That way, the following code could work, because it >>>>>>>>>>>>>>>> wouldn't throw the "unrecognized function" error: >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> label >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> .... >>>>>>>>>>>>>>>> .... >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> What do you think about that approach? >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> Meletis >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> On Friday, November 13, 2015 at 11:38:15 PM UTC+2, Martijn van de Rijdt wrote: >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> Hi, >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> I'd like to ask for a clarification about the current() >>>>>>>>>>>>>>>>> node in an XPath predicate. It seems JR considers this to be the >>>>>>>>>>>>>>>>> *parent* node of the current question. (Thanks to Chris >>>>>>>>>>>>>>>>> for sharing this.) >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> E.g. in this question below, current() equates to >>>>>>>>>>>>>>>>> /data/group and not to /data/group/options. I would have expected it to be >>>>>>>>>>>>>>>>> the latter. >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> .... >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> label >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> .... >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> .... >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> Is this by design? Or is it a bug? >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> We ran into this by realizing that the seemingly more >>>>>>>>>>>>>>>>> intuitive predicate [name = current()*/../../*crop] is >>>>>>>>>>>>>>>>> not working. >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> See this form >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> that illustrates the behaviour nicely. If current() is used in a >>>>>>>>>>>>>>>>> calculation (even if it makes no sense to do so) this extra step down is >>>>>>>>>>>>>>>>> not taken, but in a predicate it is. So it seems the behaviour is not >>>>>>>>>>>>>>>>> related to the current() function but to how predicates are handled in JR. >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> Thanks, >>>>>>>>>>>>>>>>> Martijn >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> -- >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> -- >>>>>>>>>>>>>>> Mitch Sundt >>>>>>>>>>>>>>> Software Engineer >>>>>>>>>>>>>>> University of Washington >>>>>>>>>>>>>>> mitche...@gmail.com >>>>>>>>>>>>>>> >>>>>>>>>>>>>> -- >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> -- >>>>>>>>>>>>> Mitch Sundt >>>>>>>>>>>>> Software Engineer >>>>>>>>>>>>> University of Washington >>>>>>>>>>>>> mitche...@gmail.com >>>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> -- >>>>>>>>>>>> Mitch Sundt >>>>>>>>>>>> Software Engineer >>>>>>>>>>>> University of Washington >>>>>>>>>>>> mitche...@gmail.com >>>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> -- >>>>>>>>>>> Mitch Sundt >>>>>>>>>>> Software Engineer >>>>>>>>>>> University of Washington >>>>>>>>>>> mitche...@gmail.com >>>>>>>>>>> >>>>>>>>>> -- >>>>>>>>>> You received this message because you are subscribed to the >>>>>>>>>> Google Groups "ODK Developers" group. >>>>>>>>>> To unsubscribe from this group and stop receiving emails from it, >>>>>>>>>> send an email to >>>>>>>>>> opendatakit-developers+unsubscribe@googlegroups.com. >>>>>>>>>> For more options, visit https://groups.google.com/d/optout. >>>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> -- >>>>>>>>> Mitch Sundt >>>>>>>>> Software Engineer >>>>>>>>> University of Washington >>>>>>>>> mitche...@gmail.com >>>>>>>>> >>>>>>>> -- >>>>>>> You received this message because you are subscribed to the Google >>>>>>> Groups "ODK Developers" group. >>>>>>> To unsubscribe from this group and stop receiving emails from it, >>>>>>> send an email to opendatakit-developers+unsubscribe@googlegroups.com >>>>>>> . >>>>>>> For more options, visit https://groups.google.com/d/optout. >>>>>>> >>>>>> -- >>>>>> You received this message because you are subscribed to the Google >>>>>> Groups "ODK Developers" group. >>>>>> To unsubscribe from this group and stop receiving emails from it, >>>>>> send an email to opendatakit-developers+unsubscribe@googlegroups.com >>>>>> . >>>>>> For more options, visit https://groups.google.com/d/optout. >>>>>> >>>>> >>>>> >>>>> >>>>> -- >>>>> Mitch Sundt >>>>> Software Engineer >>>>> University of Washington >>>>> mitche...@gmail.com >>>>> >>>> >>>> >>>> >>>> -- >>>> Mitch Sundt >>>> Software Engineer >>>> University of Washington >>>> mitche...@gmail.com >>>> >>>> -- >>>> You received this message because you are subscribed to the Google >>>> Groups "ODK Developers" group. >>>> To unsubscribe from this group and stop receiving emails from it, send >>>> an email to opendatakit-developers+unsubscribe@googlegroups.com >>>> . >>>> For more options, visit https://groups.google.com/d/optout. >>>> >>> -- >>> You received this message because you are subscribed to a topic in the >>> Google Groups "ODK Developers" group. >>> To unsubscribe from this topic, visit >>> https://groups.google.com/d/topic/opendatakit-developers/nN8HTA5pUIM/unsubscribe >>> . >>> To unsubscribe from this group and all its topics, send an email to >>> opendatakit-developers+unsubscribe@googlegroups.com . >>> For more options, visit https://groups.google.com/d/optout. >>> >> >> >> -- >> *Revolutionizing data collection since 2012.* >> >> Enketo | LinkedIn >> | GitHub >> | Twitter >> | Blog >> >> -- >> You received this message because you are subscribed to the Google Groups >> "ODK Developers" group. >> To unsubscribe from this group and stop receiving emails from it, send an >> email to opendatakit-developers+unsubscribe@googlegroups.com >> . >> For more options, visit https://groups.google.com/d/optout. >> > > > > -- > Mitch Sundt > Software Engineer > University of Washington > mitche...@gmail.com >

Having a number of discrete parameters that are overridable on the settings
sheet makes sense to me (like "current_references_parent" or perhaps
"current_always_field"). We could then add model parameters and change the
pyxform default over time, but users could always override on their
settings sheets as needed for specific compatibility.

For example, our SurveyCTO pyxform for 2.10 (we've given up on 2.01 now)
could default to current_always_field=false, which would mean that JR would
be backward-compatible and pyxform would emit relative paths for the
choice_filter column assuming that current() references the parent. This
would break all choice_filters in Enketo, but our Enketo users could
set current_always_field=true in their settings sheets.

For that to work, though, we'd also have to add support
for current_always_field=true in JR. We could do that as Meletis has
proposed, which may have other side effects -- we'll have to see (e.g., for
itemset copying).

Alternatively, we could try to convince Martijn to
support current_always_field=false, which is likely trivial. If he would do
that, then we could push JR's support for current_always_field=true to when
we could figure out how to do it safely. Also, it would mean producing a
version of Enketo that supports relative paths in choice filters in a
manner compatible with ODK Collect. Ona, Kobo, and other Enketo users could
rejoice, able to produce forms that use current() (like those that
reference same-repeat values) in their choice_filters. That boon to users
has to be worth something, even if a principle is slightly violated
(supporting something outside the XPath spec).

Chris

··· On Tue, Nov 24, 2015 at 8:43 PM Alex Dorey wrote:

Hi, Just catching up on the discussion. Sorry if I'm a bit behind in my
understanding of things.

Regarding the pyxform changes--

  1. Instead of the date, could we have the configuration parameter describe
    the specific change?

So instead of:

revision="2015-11-24"

we specify something like:

current_references_parent=True (default to False with the next pyxform
version bump)

  1. Could we put this configuration into the xlsform's settings sheet? I
    think this would be easier to test and configure. We could also add a way
    to override the setting with an extra argument to the CLI or xls2json call.

On Tuesday, November 24, 2015 at 3:26:18 PM UTC-5, Mitch wrote:

I agree. Having just delved into the itemset code, there are a number of
issues that would need to be cleaned up.

What I would propose is to leave JR / ODK Collect broken as-is. This
will be future work.

The python code will then be the focus of the tweaks.

(1) change:

json_survey = xls2json.workbook_to_json(json.loads(request.POST['workbookJson']),
form_name=form_name, warnings=warningsList)

to add an optional revision argument:

json_survey = xls2json.workbook_to_json(json.loads(request.POST['workbookJson']),
form_name=form_name, warnings=warningsList*, revision='2015-11-24')*

(2) change that implementation to:

(2a) store the revision string somewhere in the returned json object.

(2b) expand ${fieldname} with the new relative path expansion only if the
passed-in revision is >= '2015-11-24' ; otherwise, use the absolute path
expansion.

(3) change:

create_survey_element_from_dict()

to emit the revision value as an attribute on the element.

===============
The JR itemset bug will be fixed at a later date.

When that is fixed, we can introduce a new revision string for that
behavior change.

i.e., once we have the revision string moving through the system, we can
begin to support non-backward-compatible changes to behaviors in XLSForm
and JR form processing.

For the time being, the ODK XLSForm converter will not supply a revision
argument, and not make use of this pyxform enhancement. Once the JR fix is
done, we will then support that new revision and the current behavior,
defaulting to the new behavior, but supporting users in generating the old
behavior.

Mitch

On Tue, Nov 24, 2015 at 8:44 AM, Martijn van de Rijdt mar...@enketo.org wrote:

I appreciate the creative solution. It does sound like it will become a

bit of a headache to manage and explain to users though. Whether it's worth
it depends on how many active forms are out there that rely on the bug. We
should get Ona's and KoBo's input on this too, I think.

From Enketo's side, relative paths were never supported in
choice-filters (to my embarrassment and surprise - which led to my
question). As a result, there is no significant issue with backwards
compatibility in Enketo once it is fixed in JR. Not sure how meaningful
this is to ODK Collect, but no Enketo user has ever reported this lack of
support. I think to date it's been too hard to craft such choice-filters.
Because of the way Enketo is integrated into KoBo, Ona, Formhub, I'd be
surprised if there were a significant amount of those users using relative
choice-filters.

I do indeed feel strongly about following the XForms 2.0 spec here. I
think relative path support is going to be very useful in the future, and I
agree that we need to look at the bigger picture and do this correctly, so
it won't catch up with us.

On Tue, Nov 24, 2015 at 4:24 AM, Christopher Robert <cro...@surveycto.com wrote:

Okay, well, this is turning into a bigger and bigger project and we may

well miss the window for our 2.01 maintenance release (and/or make the
changes so extensive and scary that we cannot in good conscience include
them in a maintenance release).

However, your idea for making JR sensitive to the revision value
emitted by pyxform opens a new possibility: we could release the pyxform
change with 2.01 so that users can begin to benefit from the relative paths
(relying on today's current() behavior), then assume that when JR is
"fixed" it will only be fixed for XML emitted by newer pyxforms that would
have updated their paths to account for the fix (using your revision-value
idea). The primary drawback to this approach is that we would basically
break all cascading-select filter expressions for SurveyCTO+Enketo users
(since Martijn clearly wants to implement the "fixed" version of current()
rather than the JavaRosa-compatible version of current()).

Meletis also found another issue relating to the evaluation context in
jr:choice-name() (
https://github.com/opendatakit/opendatakit/issues/1180), so we have
that to contend with as well. If we can't safely resolve the context for
jr:choice-name(), then that would present another major drawback and
further lessen our enthusiasm for this entire adventure.

Best,

Chris

On Mon, Nov 23, 2015 at 5:45 PM Mitch Sundt mitche...@gmail.com wrote:

And, if it isn't clear, to preserve backward-compatible operations,

XLSForm should not expand ${fieldname} with a relative path unless the
revision value is the newer one; if the revision is not specified, it would
emit absolute paths like it currently does.

I also switched to use a date value for the revision value
("2015-11-23") which allows simple lexical comparisons.

On Mon, Nov 23, 2015 at 1:03 PM, Mitch Sundt mitche...@gmail.com wrote:

w.r.t. backward compatibility

I too am concerned.

A proposed solution:

  1. Add a command line argument or other mechanism to request that
    XLSForm inject a 'revision="2015-11-23"' attribute into the
    element. e.g.,

This is used to alter the behavior of the XForms engine to
accommodate backward-compatible bug behaviors.

We can't identify the bugs (e.g, 'legacy="itemset-parent"'), since
that would break older forms run on the new ODK Collect releases.
We have to advance the revision code as bugs are fixed.

I think an integer value is fine for this.

I don't anticipate making a lot of revisions, and I think the
revisions would be sequentially applied (you can't pick-and-choose what
bugs you include).

In the future, this would allow us to re-do the date-time behavior.

  1. Modify the XLSForm tools (e.g., the online form generator) to
    enable the user to select what revision they want to emit (much like we did
    last year with the JR form eval choice setting within ODK Collect). We can
    default to emitting the latest revision if the user does not make a choice.

  2. Augment JR to look for this revision attribute and store/restore
    it in the FormDef binary object.

Based upon this value, alter the behavior of the JR itemset
evaluation to fix the path bug.

  1. Modify ODK Validate to scan for syntax that may be impacted by a
    revision (to advise whether to change something). Not sure how complex
    that might be. May not be always possible?

w.r.t. impacts of change

Thank you Meletis for researching when this change occurred.

I will have to ponder what effect the parent/node change would have
on itemset copying. Hadn't thought that there was any relationship between
the two, but perhaps there is.

Yaw also reported that the Cascading Select itemsets on
opendatakit.appspot.com are not working with the 1.4.7 rev 1053
release (and they work but give odd 'null' resolutions on 1057). This is in
a related area of this same code, so I will hopefully gain a better
understanding of dynamic itemsets as I uncover the causes of that problem.


Mitch

On Mon, Nov 23, 2015 at 5:59 AM, Christopher Robert < cro...@surveycto.com> wrote:

So just to be clear, the "fix" we're testing here reverts the entire

context within that populateDynamicChoices() function, to be the field
instead of the parent -- so the effects are potentially much wider than
just what current() returns. Potentially anything and everything that
depends on context can change now, since the context will shift from parent
to field.

This means that we revert a change from 5 years ago and alter the
context from now on. Neither Meletis nor I feel terribly confident that we
can entirely predict the consequences to backward compatibility. We're
testing the things we can think to test, but honestly it's XML users in the
broader ODK community who face the greatest risks here; SurveyCTO users by
and large use XLSForm and thus the potential XPath syntax consequences are
sharply limited.

Are you guys very sure that you want to take this path? I can tell
you now that if we make this change and then there is some outcry after the
next ODK Collect release, SurveyCTO Collect will very likely stick with the
new change (since we will have changed pyxform and we'll then have too many
backward-compatibility concerns to then reverse the change).

Thanks,

Chris

On Mon, Nov 23, 2015 at 5:23 AM mel...@surveycto.com wrote:

Just for the history's sake, I found the time where the behavior
changed. It was at March 11, 2010:

https://bitbucket.org/m.sundt/javarosa/commits/b41875b2c23c9a21a868a5d2dc0ffc1f6b5b0b65#Lcore/src/org/javarosa/core/model/FormDef.javaT635

Take a look only in the changes of the "populateDynamicChoices"
method. The context reference used to be the "current" field, but after the
changes above it started be the field's parent (by adding getParent()).
Perhaps there was a reason for that change, but perhaps not, but I think
all of us agree that this should be fixed, even after 5 and a half years.

Meletis

On Monday, November 23, 2015 at 12:08:47 PM UTC+2, mel...@surveycto.com wrote:

Hi Mitch,

Please take a look at the code comment in line 31 here:

https://bitbucket.org/m.sundt/javarosa/src/bf17fb749b74e53915f086ca3827c3c6fe73fd07/core/src/org/javarosa/core/model/ItemsetBinding.java?at=default&fileviewer=file-view-default#ItemsetBinding.java-31

It clearly says: "ref of the control parent (group/formdef) of
itemset question
"

So, for some reason, they wanted it to point to the parent. I
also see that when the dynamic choices are populated, the evaluation
context is wrapped around that parent (contextRef object), that's why
current() refers to the parent group when evaluating itemsets nodesets.

I will run a few tests by changing the contextRef field value, I
just wanted to mention that in case you remember a reason for that code
comment.

Meletis

On Friday, November 20, 2015 at 8:49:12 PM UTC+2, Mitch wrote:

Yes, I think it should be a one-line change; just don't know
where.

Mitch

(I do jar-updates over Christmas / New-Years. Tedious, but
doesn't require a whole lot of attention.)

On Fri, Nov 20, 2015 at 10:36 AM, mel...@surveycto.com wrote:

Hi Mitch,

Thank you for all the details, I guess I was missing some of the
context of the original issue.

We will give it a shot to see if we can come up with a few ideas
how to fix the evaluation content of the itemset nodeset. I just hope it
will not be like last year with the "form processing logic" in JR,
otherwise I believe that every Christmas will be a JR Christmas. :slight_smile:

Meletis

On Friday, November 20, 2015 at 7:29:14 PM UTC+2, Mitch wrote:

The form was attached in one of my earlier replies.

On Fri, Nov 20, 2015 at 9:27 AM, Mitch Sundt < mitche...@gmail.com> wrote:

And you see this in constraint expressions where you write:

. > 6

I.e., current()/. should refer to the field, and not the group
containing the field, as it currently does when evaluating expressions in
the itemset nodeset.

On Fri, Nov 20, 2015 at 9:26 AM, Mitch Sundt < mitche...@gmail.com> wrote:

The issue is that there is a discrepancy in the meaning of
'.' across the different contexts of: calculate, relevant, , and
these itemset nodeset expressions.

In other usages, you need to reference this other field with
current()/../filterfield ; but in the nodeset expression, you would
reference it with current()/./filterfield

I.e., the notion of '.' is different when used within a
nodeset expression and when used within a constraint expression or an
field.

See the attached NBiggestOfSet form. For the label, it has:

    <select1 ref="/NBiggestOfSet/education/edu_level">
      <label>What is <output value="

current()/../referred_attendee_name"/>'s level of
education?
...

For a constraint, it has:

These each use the current()/../ to reference the group (
/NBiggesetOfSet/education ).

But, within the nodeset definition (if this form used one),
you would need to use current()/./referred_attendee_name
to reference a field in that group.

On Fri, Nov 20, 2015 at 8:22 AM, mel...@surveycto.com wrote:

Hi Mitch,

I think I'm missing something but I think that nothing is
broken and that the only issue is that the
expression current()/../filterfield in your example above should be just
changed to current()/filterfield. If you make that change, everything
should work because it will be resolved to:

/data/group1/filterfield

Why should a "../" be present there? We have used
current()/somefield successfully in one of our test forms and it works
fine. And I think it would also work great in your example. Wouldn't it? Or
am I totally out of context?

Meletis

On Friday, November 20, 2015 at 5:35:16 PM UTC+2, Mitch wrote:

Hi Meletis,

That's not the issue --

The issue is that if you use a filter condition in the
nodeset specification of a field "myfield", within that nodeset
specification "current()/.." refers to the parent of the group enclosing
the "myfield" field. It does not refer to the group enclosing the
"myfield" field.

So if you have

a 4 fish b 14 fish ...

and if you have:

This will currently be broken. Because
current()/../filterfield,
when evaluated within the itemset nodeset specification, is
resolved to:

/data/filterfield

instead of

/data/group1/filterfield

On Fri, Nov 20, 2015 at 1:58 AM, mel...@surveycto.com wrote:

Well, I think there is a fundamental issue here regarding
current() syntax. Please, let me remind you how JR works with "current()":

  1. The expression "current()" looks like a function but it
    is not treated like one; it is not handled by the XPathFuncExpr class, so,
    if you try to use it alone you will get an error that the "function" is not
    supported. In other words, you cannot use it alone in an expression in
    order to refer to the "current" field.

  2. When an expression includes "current()", then it ALWAYS
    needs to be used in an XPath, actually, the expression needs to BE an XPath
    expression, so that it can be handled with the XPathPathExpr class. For
    that reason the expression NEEDS to contain at least one "/".

  3. Taking 1 and 2 into consideration, the ONLY way to
    refer to the "current" field (named "somefield") is to use "current()/." or
    "current()/somefield". So, I looks like that the original implementation
    was "okay" in the sense that the "current()" should only be used in XPaths.

If we wanted to have "current()" refer to the current
field, I believe we should introduce a new function handler in
XPathFuncExpr class. Please let me know if this would work:

A. Whenever the "current()" expression is used inside an
XPath, then the current code would be used as it is used right now
(maintaining backwards compatibility), so that it will still be referring
to the enclosing group, not the field itself.
B. Whenever the "current()" expression is used inside a
non-XPath, then the new code would be used (in XPathFuncExpr) and that
would be evaluated in a way that the current field would be
returned/referenced, NOT the enclosing group.

That way, the following code could work, because it
wouldn't throw the "unrecognized function" error:

label .... ....

What do you think about that approach?

Meletis

On Friday, November 13, 2015 at 11:38:15 PM UTC+2, Martijn van de Rijdt wrote:

Hi,

I'd like to ask for a clarification about the current()
node in an XPath predicate. It seems JR considers this to be the
parent node of the current question. (Thanks to Chris
for sharing this.)

E.g. in this question below, current() equates to
/data/group and not to /data/group/options. I would have expected it to be
the latter.

....
    <label>label</label>

    <itemset nodeset="instance('crop_list')/root/item[name

= current()*/../*crop]">

         ....

         ....

    </itemset>

Is this by design? Or is it a bug?

We ran into this by realizing that the seemingly more
intuitive predicate [name = current()*/../../*crop] is
not working.

See this form
https://docs.google.com/spreadsheets/d/1A74yEHRxQF1zbIcQSrwvZydggUU7m5T3ZAnlR6_TLA4/edit#gid=0
that illustrates the behaviour nicely. If current() is used in a
calculation (even if it makes no sense to do so) this extra step down is
not taken, but in a predicate it is. So it seems the behaviour is not
related to the current() function but to how predicates are handled in JR.

Thanks,
Martijn

--

--
Mitch Sundt
Software Engineer
University of Washington
mitche...@gmail.com

--

--
Mitch Sundt
Software Engineer
University of Washington
mitche...@gmail.com

--
Mitch Sundt
Software Engineer
University of Washington
mitche...@gmail.com

--
Mitch Sundt
Software Engineer
University of Washington
mitche...@gmail.com

--
You received this message because you are subscribed to the
Google Groups "ODK Developers" group.
To unsubscribe from this group and stop receiving emails from
it, send an email to
opendatakit-developers+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--
Mitch Sundt
Software Engineer
University of Washington
mitche...@gmail.com

--
You received this message because you are subscribed to the Google
Groups "ODK Developers" group.
To unsubscribe from this group and stop receiving emails from it,
send an email to
opendatakit-developers+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google
Groups "ODK Developers" group.
To unsubscribe from this group and stop receiving emails from it,
send an email to opendatakit-developers+unsubscribe@googlegroups.com
.
For more options, visit https://groups.google.com/d/optout.

--

Mitch Sundt

Software Engineer
University of Washington
mitche...@gmail.com

--
Mitch Sundt
Software Engineer
University of Washington
mitche...@gmail.com

--

You received this message because you are subscribed to the Google
Groups "ODK Developers" group.
To unsubscribe from this group and stop receiving emails from it, send
an email to opendatakit-developers+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--

You received this message because you are subscribed to a topic in the

Google Groups "ODK Developers" group.
To unsubscribe from this topic, visit
https://groups.google.com/d/topic/opendatakit-developers/nN8HTA5pUIM/unsubscribe
.
To unsubscribe from this group and all its topics, send an email to
opendatakit-developers+unsubscribe@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--
Revolutionizing data collection since 2012.

Enketo https://enketo.org/ | LinkedIn
http://www.linkedin.com/company/enketo-llc | GitHub
https://github.com/enketo | Twitter https://twitter.com/enketo
| Blog http://blog.enketo.org/

--
You received this message because you are subscribed to the Google
Groups "ODK Developers" group.
To unsubscribe from this group and stop receiving emails from it, send
an email to opendatakit-developers+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--

Mitch Sundt

Software Engineer
University of Washington

mitche...@gmail.com

--
You received this message because you are subscribed to the Google Groups
"ODK Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to opendatakit-developers+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

All this sounds fine to me, but wanted to raise some potential issues.
Discrete parameters also means folks can pick and choose what behavior
they want and it's hard to move the entire community forward that way.
Also, once you get past two or three, you end up with a settings sheet
of lots of flags to toggle and that isn't a great user experience.

Yaw

··· -- Need ODK consultants? Nafundi provides form design, server setup, in-field training, and software development for ODK. Go to https://nafundi.com to get started.

On Sun, Nov 29, 2015 at 6:47 PM, Christopher Robert crobert@surveycto.com wrote:

Having a number of discrete parameters that are overridable on the settings
sheet makes sense to me (like "current_references_parent" or perhaps
"current_always_field"). We could then add model parameters and change the
pyxform default over time, but users could always override on their settings
sheets as needed for specific compatibility.

For example, our SurveyCTO pyxform for 2.10 (we've given up on 2.01 now)
could default to current_always_field=false, which would mean that JR would
be backward-compatible and pyxform would emit relative paths for the
choice_filter column assuming that current() references the parent. This
would break all choice_filters in Enketo, but our Enketo users could set
current_always_field=true in their settings sheets.

For that to work, though, we'd also have to add support for
current_always_field=true in JR. We could do that as Meletis has proposed,
which may have other side effects -- we'll have to see (e.g., for itemset
copying).

Alternatively, we could try to convince Martijn to support
current_always_field=false, which is likely trivial. If he would do that,
then we could push JR's support for current_always_field=true to when we
could figure out how to do it safely. Also, it would mean producing a
version of Enketo that supports relative paths in choice filters in a manner
compatible with ODK Collect. Ona, Kobo, and other Enketo users could
rejoice, able to produce forms that use current() (like those that reference
same-repeat values) in their choice_filters. That boon to users has to be
worth something, even if a principle is slightly violated (supporting
something outside the XPath spec).

Chris

On Tue, Nov 24, 2015 at 8:43 PM Alex Dorey dorey415@gmail.com wrote:

Hi, Just catching up on the discussion. Sorry if I'm a bit behind in my
understanding of things.

Regarding the pyxform changes--

  1. Instead of the date, could we have the configuration parameter describe
    the specific change?

So instead of:

revision="2015-11-24"

we specify something like:

current_references_parent=True (default to False with the next pyxform
version bump)

  1. Could we put this configuration into the xlsform's settings sheet? I
    think this would be easier to test and configure. We could also add a way to
    override the setting with an extra argument to the CLI or xls2json call.

On Tuesday, November 24, 2015 at 3:26:18 PM UTC-5, Mitch wrote:

I agree. Having just delved into the itemset code, there are a number of
issues that would need to be cleaned up.

What I would propose is to leave JR / ODK Collect broken as-is. This
will be future work.

The python code will then be the focus of the tweaks.

(1) change:

json_survey =
xls2json.workbook_to_json(json.loads(request.POST['workbookJson']),
form_name=form_name, warnings=warningsList)

to add an optional revision argument:

json_survey =
xls2json.workbook_to_json(json.loads(request.POST['workbookJson']),
form_name=form_name, warnings=warningsList, revision='2015-11-24')

(2) change that implementation to:

(2a) store the revision string somewhere in the returned json object.

(2b) expand ${fieldname} with the new relative path expansion only if the
passed-in revision is >= '2015-11-24' ; otherwise, use the absolute path
expansion.

(3) change:

create_survey_element_from_dict()

to emit the revision value as an attribute on the element.

===============
The JR itemset bug will be fixed at a later date.

When that is fixed, we can introduce a new revision string for that
behavior change.

i.e., once we have the revision string moving through the system, we can
begin to support non-backward-compatible changes to behaviors in XLSForm and
JR form processing.

For the time being, the ODK XLSForm converter will not supply a revision
argument, and not make use of this pyxform enhancement. Once the JR fix is
done, we will then support that new revision and the current behavior,
defaulting to the new behavior, but supporting users in generating the old
behavior.

Mitch

On Tue, Nov 24, 2015 at 8:44 AM, Martijn van de Rijdt mar...@enketo.org wrote:

I appreciate the creative solution. It does sound like it will become a
bit of a headache to manage and explain to users though. Whether it's worth
it depends on how many active forms are out there that rely on the bug. We
should get Ona's and KoBo's input on this too, I think.

From Enketo's side, relative paths were never supported in
choice-filters (to my embarrassment and surprise - which led to my
question). As a result, there is no significant issue with backwards
compatibility in Enketo once it is fixed in JR. Not sure how meaningful this
is to ODK Collect, but no Enketo user has ever reported this lack of
support. I think to date it's been too hard to craft such choice-filters.
Because of the way Enketo is integrated into KoBo, Ona, Formhub, I'd be
surprised if there were a significant amount of those users using relative
choice-filters.

I do indeed feel strongly about following the XForms 2.0 spec here. I
think relative path support is going to be very useful in the future, and I
agree that we need to look at the bigger picture and do this correctly, so
it won't catch up with us.

On Tue, Nov 24, 2015 at 4:24 AM, Christopher Robert cro...@surveycto.com wrote:

Okay, well, this is turning into a bigger and bigger project and we may
well miss the window for our 2.01 maintenance release (and/or make the
changes so extensive and scary that we cannot in good conscience include
them in a maintenance release).

However, your idea for making JR sensitive to the revision value
emitted by pyxform opens a new possibility: we could release the pyxform
change with 2.01 so that users can begin to benefit from the relative paths
(relying on today's current() behavior), then assume that when JR is "fixed"
it will only be fixed for XML emitted by newer pyxforms that would have
updated their paths to account for the fix (using your revision-value idea).
The primary drawback to this approach is that we would basically break all
cascading-select filter expressions for SurveyCTO+Enketo users (since
Martijn clearly wants to implement the "fixed" version of current() rather
than the JavaRosa-compatible version of current()).

Meletis also found another issue relating to the evaluation context in
jr:choice-name() (https://github.com/opendatakit/opendatakit/issues/1180),
so we have that to contend with as well. If we can't safely resolve the
context for jr:choice-name(), then that would present another major drawback
and further lessen our enthusiasm for this entire adventure.

Best,

Chris

On Mon, Nov 23, 2015 at 5:45 PM Mitch Sundt mitche...@gmail.com wrote:

And, if it isn't clear, to preserve backward-compatible operations,
XLSForm should not expand ${fieldname} with a relative path unless the
revision value is the newer one; if the revision is not specified, it would
emit absolute paths like it currently does.

I also switched to use a date value for the revision value
("2015-11-23") which allows simple lexical comparisons.

On Mon, Nov 23, 2015 at 1:03 PM, Mitch Sundt mitche...@gmail.com wrote:

w.r.t. backward compatibility

I too am concerned.

A proposed solution:

  1. Add a command line argument or other mechanism to request that
    XLSForm inject a 'revision="2015-11-23"' attribute into the element.
    e.g.,

This is used to alter the behavior of the XForms engine to
accommodate backward-compatible bug behaviors.

We can't identify the bugs (e.g, 'legacy="itemset-parent"'), since
that would break older forms run on the new ODK Collect releases.
We have to advance the revision code as bugs are fixed.

I think an integer value is fine for this.

I don't anticipate making a lot of revisions, and I think the
revisions would be sequentially applied (you can't pick-and-choose what bugs
you include).

In the future, this would allow us to re-do the date-time behavior.

  1. Modify the XLSForm tools (e.g., the online form generator) to
    enable the user to select what revision they want to emit (much like we did
    last year with the JR form eval choice setting within ODK Collect). We can
    default to emitting the latest revision if the user does not make a choice.

  2. Augment JR to look for this revision attribute and store/restore
    it in the FormDef binary object.

Based upon this value, alter the behavior of the JR itemset
evaluation to fix the path bug.

  1. Modify ODK Validate to scan for syntax that may be impacted by a
    revision (to advise whether to change something). Not sure how complex that
    might be. May not be always possible?

w.r.t. impacts of change

Thank you Meletis for researching when this change occurred.

I will have to ponder what effect the parent/node change would have
on itemset copying. Hadn't thought that there was any relationship between
the two, but perhaps there is.

Yaw also reported that the Cascading Select itemsets on
opendatakit.appspot.com are not working with the 1.4.7 rev 1053 release (and
they work but give odd 'null' resolutions on 1057). This is in a related
area of this same code, so I will hopefully gain a better understanding of
dynamic itemsets as I uncover the causes of that problem.


Mitch

On Mon, Nov 23, 2015 at 5:59 AM, Christopher Robert cro...@surveycto.com wrote:

So just to be clear, the "fix" we're testing here reverts the entire
context within that populateDynamicChoices() function, to be the field
instead of the parent -- so the effects are potentially much wider than just
what current() returns. Potentially anything and everything that depends on
context can change now, since the context will shift from parent to field.

This means that we revert a change from 5 years ago and alter the
context from now on. Neither Meletis nor I feel terribly confident that we
can entirely predict the consequences to backward compatibility. We're
testing the things we can think to test, but honestly it's XML users in the
broader ODK community who face the greatest risks here; SurveyCTO users by
and large use XLSForm and thus the potential XPath syntax consequences are
sharply limited.

Are you guys very sure that you want to take this path? I can tell
you now that if we make this change and then there is some outcry after the
next ODK Collect release, SurveyCTO Collect will very likely stick with the
new change (since we will have changed pyxform and we'll then have too many
backward-compatibility concerns to then reverse the change).

Thanks,

Chris

On Mon, Nov 23, 2015 at 5:23 AM mel...@surveycto.com wrote:

Just for the history's sake, I found the time where the behavior
changed. It was at March 11, 2010:

https://bitbucket.org/m.sundt/javarosa/commits/b41875b2c23c9a21a868a5d2dc0ffc1f6b5b0b65#Lcore/src/org/javarosa/core/model/FormDef.javaT635

Take a look only in the changes of the "populateDynamicChoices"
method. The context reference used to be the "current" field, but after the
changes above it started be the field's parent (by adding getParent()).
Perhaps there was a reason for that change, but perhaps not, but I think all
of us agree that this should be fixed, even after 5 and a half years.

Meletis

On Monday, November 23, 2015 at 12:08:47 PM UTC+2, mel...@surveycto.com wrote:

Hi Mitch,

Please take a look at the code comment in line 31 here:

https://bitbucket.org/m.sundt/javarosa/src/bf17fb749b74e53915f086ca3827c3c6fe73fd07/core/src/org/javarosa/core/model/ItemsetBinding.java?at=default&fileviewer=file-view-default#ItemsetBinding.java-31

It clearly says: "ref of the control parent (group/formdef) of
itemset question"

So, for some reason, they wanted it to point to the parent. I also
see that when the dynamic choices are populated, the evaluation context is
wrapped around that parent (contextRef object), that's why current() refers
to the parent group when evaluating itemsets nodesets.

I will run a few tests by changing the contextRef field value, I
just wanted to mention that in case you remember a reason for that code
comment.

Meletis

On Friday, November 20, 2015 at 8:49:12 PM UTC+2, Mitch wrote:

Yes, I think it should be a one-line change; just don't know
where.

Mitch

(I do jar-updates over Christmas / New-Years. Tedious, but
doesn't require a whole lot of attention.)

On Fri, Nov 20, 2015 at 10:36 AM, mel...@surveycto.com wrote:

Hi Mitch,

Thank you for all the details, I guess I was missing some of the
context of the original issue.

We will give it a shot to see if we can come up with a few ideas
how to fix the evaluation content of the itemset nodeset. I just hope it
will not be like last year with the "form processing logic" in JR, otherwise
I believe that every Christmas will be a JR Christmas. :slight_smile:

Meletis

On Friday, November 20, 2015 at 7:29:14 PM UTC+2, Mitch wrote:

The form was attached in one of my earlier replies.

On Fri, Nov 20, 2015 at 9:27 AM, Mitch Sundt mitche...@gmail.com wrote:

And you see this in constraint expressions where you write:

. > 6

I.e., current()/. should refer to the field, and not the group
containing the field, as it currently does when evaluating expressions in
the itemset nodeset.

On Fri, Nov 20, 2015 at 9:26 AM, Mitch Sundt mitche...@gmail.com wrote:

The issue is that there is a discrepancy in the meaning of
'.' across the different contexts of: calculate, relevant, , and
these itemset nodeset expressions.

In other usages, you need to reference this other field with
current()/../filterfield ; but in the nodeset expression, you would
reference it with current()/./filterfield

I.e., the notion of '.' is different when used within a
nodeset expression and when used within a constraint expression or an
field.

See the attached NBiggestOfSet form. For the label, it has:

    <select1 ref="/NBiggestOfSet/education/edu_level">
      <label>What is <output

value="current()/../referred_attendee_name"/>'s level of education?
...

For a constraint, it has:

These each use the current()/../ to reference the group (
/NBiggesetOfSet/education ).

But, within the nodeset definition (if this form used one),
you would need to use current()/./referred_attendee_name to reference a
field in that group.

On Fri, Nov 20, 2015 at 8:22 AM, mel...@surveycto.com wrote:

Hi Mitch,

I think I'm missing something but I think that nothing is
broken and that the only issue is that the expression
current()/../filterfield in your example above should be just changed to
current()/filterfield. If you make that change, everything should work
because it will be resolved to:

/data/group1/filterfield

Why should a "../" be present there? We have used
current()/somefield successfully in one of our test forms and it works fine.
And I think it would also work great in your example. Wouldn't it? Or am I
totally out of context?

Meletis

On Friday, November 20, 2015 at 5:35:16 PM UTC+2, Mitch wrote:

Hi Meletis,

That's not the issue --

The issue is that if you use a filter condition in the
nodeset specification of a field "myfield", within that nodeset
specification "current()/.." refers to the parent of the group enclosing
the "myfield" field. It does not refer to the group enclosing the "myfield"
field.

So if you have

a 4 fish b 14 fish ...

and if you have:

This will currently be broken. Because
current()/../filterfield,
when evaluated within the itemset nodeset specification, is
resolved to:

/data/filterfield

instead of

/data/group1/filterfield

On Fri, Nov 20, 2015 at 1:58 AM, mel...@surveycto.com wrote:

Well, I think there is a fundamental issue here regarding
current() syntax. Please, let me remind you how JR works with "current()":

  1. The expression "current()" looks like a function but it
    is not treated like one; it is not handled by the XPathFuncExpr class, so,
    if you try to use it alone you will get an error that the "function" is not
    supported. In other words, you cannot use it alone in an expression in order
    to refer to the "current" field.

  2. When an expression includes "current()", then it ALWAYS
    needs to be used in an XPath, actually, the expression needs to BE an XPath
    expression, so that it can be handled with the XPathPathExpr class. For that
    reason the expression NEEDS to contain at least one "/".

  3. Taking 1 and 2 into consideration, the ONLY way to
    refer to the "current" field (named "somefield") is to use "current()/." or
    "current()/somefield". So, I looks like that the original implementation was
    "okay" in the sense that the "current()" should only be used in XPaths.

If we wanted to have "current()" refer to the current
field, I believe we should introduce a new function handler in XPathFuncExpr
class. Please let me know if this would work:

A. Whenever the "current()" expression is used inside an
XPath, then the current code would be used as it is used right now
(maintaining backwards compatibility), so that it will still be referring to
the enclosing group, not the field itself.
B. Whenever the "current()" expression is used inside a
non-XPath, then the new code would be used (in XPathFuncExpr) and that would
be evaluated in a way that the current field would be returned/referenced,
NOT the enclosing group.

That way, the following code could work, because it
wouldn't throw the "unrecognized function" error:

label .... ....

What do you think about that approach?

Meletis

On Friday, November 13, 2015 at 11:38:15 PM UTC+2, Martijn van de Rijdt wrote:

Hi,

I'd like to ask for a clarification about the current()
node in an XPath predicate. It seems JR considers this to be the parent node
of the current question. (Thanks to Chris for sharing this.)

E.g. in this question below, current() equates to
/data/group and not to /data/group/options. I would have expected it to be
the latter.

....
    <label>label</label>

    <itemset

nodeset="instance('crop_list')/root/item[name = current()/../crop]">

         ....

         ....

    </itemset>

Is this by design? Or is it a bug?

We ran into this by realizing that the seemingly more
intuitive predicate [name = current()/../../crop] is not working.

See this form that illustrates the behaviour nicely. If
current() is used in a calculation (even if it makes no sense to do so) this
extra step down is not taken, but in a predicate it is. So it seems the
behaviour is not related to the current() function but to how predicates are
handled in JR.

Thanks,
Martijn

--

--
Mitch Sundt
Software Engineer
University of Washington
mitche...@gmail.com

--

--
Mitch Sundt
Software Engineer
University of Washington
mitche...@gmail.com

--
Mitch Sundt
Software Engineer
University of Washington
mitche...@gmail.com

--
Mitch Sundt
Software Engineer
University of Washington
mitche...@gmail.com

--
You received this message because you are subscribed to the
Google Groups "ODK Developers" group.
To unsubscribe from this group and stop receiving emails from
it, send an email to opendatakit-developers+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--
Mitch Sundt
Software Engineer
University of Washington
mitche...@gmail.com

--
You received this message because you are subscribed to the Google
Groups "ODK Developers" group.
To unsubscribe from this group and stop receiving emails from it,
send an email to opendatakit-developers+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google
Groups "ODK Developers" group.
To unsubscribe from this group and stop receiving emails from it,
send an email to opendatakit-developers+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--

Mitch Sundt
Software Engineer
University of Washington
mitche...@gmail.com

--
Mitch Sundt
Software Engineer
University of Washington
mitche...@gmail.com

--
You received this message because you are subscribed to the Google
Groups "ODK Developers" group.
To unsubscribe from this group and stop receiving emails from it, send
an email to opendatakit-developers+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--

You received this message because you are subscribed to a topic in the
Google Groups "ODK Developers" group.
To unsubscribe from this topic, visit
https://groups.google.com/d/topic/opendatakit-developers/nN8HTA5pUIM/unsubscribe.
To unsubscribe from this group and all its topics, send an email to
opendatakit-developers+unsubscribe@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--
Revolutionizing data collection since 2012.

Enketo | LinkedIn | GitHub | Twitter | Blog

--
You received this message because you are subscribed to the Google
Groups "ODK Developers" group.
To unsubscribe from this group and stop receiving emails from it, send
an email to opendatakit-developers+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--

Mitch Sundt
Software Engineer
University of Washington

mitche...@gmail.com

--
You received this message because you are subscribed to the Google Groups
"ODK Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to opendatakit-developers+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups
"ODK Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to opendatakit-developers+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

I wasn't going to suggest adding the different parameters to any of the
XLSForm templates -- so they wouldn't become cluttered with this stuff.
Really, it would just be an emergency option for somebody requiring
different compatibility for some component that was out of sync with their
other components. Ideally, nobody would actually have to use the settings
sheet to override any of these options.

To be specific, my proposal would be that everybody
support current_always_field=false today, and that this be how pyxform
today defaults. Ideally, this would include Enketo, as Collect and
SurveyCTO's Web Collect (and anything else using the ODK JavaRosa) already
support current_always_field=false as their current behavior. Then, we
figure out how to get current_always_field=true to work in JavaRosa and
phase that in, eventually changing the pyxform default to that once we
think that enough components support it.

We have thousands of devices out there using older Collect versions, even
well after their servers have been updated... so my guess is that it's some
of those users who will end up having to use the settings-sheet option to
force current_always_field=false after we've changed the default to true.

Does all of that make sense?

Chris

··· On Sun, Nov 29, 2015 at 6:48 PM Yaw Anokwa wrote:

All this sounds fine to me, but wanted to raise some potential issues.
Discrete parameters also means folks can pick and choose what behavior
they want and it's hard to move the entire community forward that way.
Also, once you get past two or three, you end up with a settings sheet
of lots of flags to toggle and that isn't a great user experience.

Yaw

Need ODK consultants? Nafundi provides form design, server setup,
in-field training, and software development for ODK. Go to
https://nafundi.com to get started.

On Sun, Nov 29, 2015 at 6:47 PM, Christopher Robert crobert@surveycto.com wrote:

Having a number of discrete parameters that are overridable on the
settings
sheet makes sense to me (like "current_references_parent" or perhaps
"current_always_field"). We could then add model parameters and change
the
pyxform default over time, but users could always override on their
settings
sheets as needed for specific compatibility.

For example, our SurveyCTO pyxform for 2.10 (we've given up on 2.01 now)
could default to current_always_field=false, which would mean that JR
would
be backward-compatible and pyxform would emit relative paths for the
choice_filter column assuming that current() references the parent. This
would break all choice_filters in Enketo, but our Enketo users could set
current_always_field=true in their settings sheets.

For that to work, though, we'd also have to add support for
current_always_field=true in JR. We could do that as Meletis has
proposed,
which may have other side effects -- we'll have to see (e.g., for itemset
copying).

Alternatively, we could try to convince Martijn to support
current_always_field=false, which is likely trivial. If he would do that,
then we could push JR's support for current_always_field=true to when we
could figure out how to do it safely. Also, it would mean producing a
version of Enketo that supports relative paths in choice filters in a
manner
compatible with ODK Collect. Ona, Kobo, and other Enketo users could
rejoice, able to produce forms that use current() (like those that
reference
same-repeat values) in their choice_filters. That boon to users has to be
worth something, even if a principle is slightly violated (supporting
something outside the XPath spec).

Chris

On Tue, Nov 24, 2015 at 8:43 PM Alex Dorey dorey415@gmail.com wrote:

Hi, Just catching up on the discussion. Sorry if I'm a bit behind in my
understanding of things.

Regarding the pyxform changes--

  1. Instead of the date, could we have the configuration parameter
    describe
    the specific change?

So instead of:

revision="2015-11-24"

we specify something like:

current_references_parent=True (default to False with the next pyxform
version bump)

  1. Could we put this configuration into the xlsform's settings sheet? I
    think this would be easier to test and configure. We could also add a
    way to
    override the setting with an extra argument to the CLI or xls2json call.

On Tuesday, November 24, 2015 at 3:26:18 PM UTC-5, Mitch wrote:

I agree. Having just delved into the itemset code, there are a number
of
issues that would need to be cleaned up.

What I would propose is to leave JR / ODK Collect broken as-is. This
will be future work.

The python code will then be the focus of the tweaks.

(1) change:

json_survey =
xls2json.workbook_to_json(json.loads(request.POST['workbookJson']),
form_name=form_name, warnings=warningsList)

to add an optional revision argument:

json_survey =
xls2json.workbook_to_json(json.loads(request.POST['workbookJson']),
form_name=form_name, warnings=warningsList, revision='2015-11-24')

(2) change that implementation to:

(2a) store the revision string somewhere in the returned json object.

(2b) expand ${fieldname} with the new relative path expansion only if
the
passed-in revision is >= '2015-11-24' ; otherwise, use the absolute
path
expansion.

(3) change:

create_survey_element_from_dict()

to emit the revision value as an attribute on the element.

===============
The JR itemset bug will be fixed at a later date.

When that is fixed, we can introduce a new revision string for that
behavior change.

i.e., once we have the revision string moving through the system, we
can
begin to support non-backward-compatible changes to behaviors in
XLSForm and
JR form processing.

For the time being, the ODK XLSForm converter will not supply a
revision
argument, and not make use of this pyxform enhancement. Once the JR
fix is
done, we will then support that new revision and the current behavior,
defaulting to the new behavior, but supporting users in generating the
old
behavior.

Mitch

On Tue, Nov 24, 2015 at 8:44 AM, Martijn van de Rijdt < mar...@enketo.org> wrote:

I appreciate the creative solution. It does sound like it will become
a
bit of a headache to manage and explain to users though. Whether it's
worth
it depends on how many active forms are out there that rely on the
bug. We
should get Ona's and KoBo's input on this too, I think.

From Enketo's side, relative paths were never supported in
choice-filters (to my embarrassment and surprise - which led to my
question). As a result, there is no significant issue with backwards
compatibility in Enketo once it is fixed in JR. Not sure how
meaningful this
is to ODK Collect, but no Enketo user has ever reported this lack of
support. I think to date it's been too hard to craft such
choice-filters.
Because of the way Enketo is integrated into KoBo, Ona, Formhub, I'd
be
surprised if there were a significant amount of those users using
relative
choice-filters.

I do indeed feel strongly about following the XForms 2.0 spec here. I
think relative path support is going to be very useful in the future,
and I
agree that we need to look at the bigger picture and do this
correctly, so
it won't catch up with us.

On Tue, Nov 24, 2015 at 4:24 AM, Christopher Robert cro...@surveycto.com wrote:

Okay, well, this is turning into a bigger and bigger project and we
may
well miss the window for our 2.01 maintenance release (and/or make
the
changes so extensive and scary that we cannot in good conscience
include
them in a maintenance release).

However, your idea for making JR sensitive to the revision value
emitted by pyxform opens a new possibility: we could release the
pyxform
change with 2.01 so that users can begin to benefit from the
relative paths
(relying on today's current() behavior), then assume that when JR is
"fixed"
it will only be fixed for XML emitted by newer pyxforms that would
have
updated their paths to account for the fix (using your
revision-value idea).
The primary drawback to this approach is that we would basically
break all
cascading-select filter expressions for SurveyCTO+Enketo users (since
Martijn clearly wants to implement the "fixed" version of current()
rather
than the JavaRosa-compatible version of current()).

Meletis also found another issue relating to the evaluation context
in
jr:choice-name() (
https://github.com/opendatakit/opendatakit/issues/1180),
so we have that to contend with as well. If we can't safely resolve
the
context for jr:choice-name(), then that would present another major
drawback
and further lessen our enthusiasm for this entire adventure.

Best,

Chris

On Mon, Nov 23, 2015 at 5:45 PM Mitch Sundt mitche...@gmail.com wrote:

And, if it isn't clear, to preserve backward-compatible operations,
XLSForm should not expand ${fieldname} with a relative path unless
the
revision value is the newer one; if the revision is not specified,
it would
emit absolute paths like it currently does.

I also switched to use a date value for the revision value
("2015-11-23") which allows simple lexical comparisons.

On Mon, Nov 23, 2015 at 1:03 PM, Mitch Sundt mitche...@gmail.com wrote:

w.r.t. backward compatibility

I too am concerned.

A proposed solution:

  1. Add a command line argument or other mechanism to request that
    XLSForm inject a 'revision="2015-11-23"' attribute into the
    element.
    e.g.,

This is used to alter the behavior of the XForms engine to
accommodate backward-compatible bug behaviors.

We can't identify the bugs (e.g, 'legacy="itemset-parent"'), since
that would break older forms run on the new ODK Collect releases.
We have to advance the revision code as bugs are fixed.

I think an integer value is fine for this.

I don't anticipate making a lot of revisions, and I think the
revisions would be sequentially applied (you can't pick-and-choose
what bugs
you include).

In the future, this would allow us to re-do the date-time behavior.

  1. Modify the XLSForm tools (e.g., the online form generator) to
    enable the user to select what revision they want to emit (much
    like we did
    last year with the JR form eval choice setting within ODK
    Collect). We can
    default to emitting the latest revision if the user does not make
    a choice.

  2. Augment JR to look for this revision attribute and store/restore
    it in the FormDef binary object.

Based upon this value, alter the behavior of the JR itemset
evaluation to fix the path bug.

  1. Modify ODK Validate to scan for syntax that may be impacted by a
    revision (to advise whether to change something). Not sure how
    complex that
    might be. May not be always possible?

w.r.t. impacts of change

Thank you Meletis for researching when this change occurred.

I will have to ponder what effect the parent/node change would have
on itemset copying. Hadn't thought that there was any
relationship between
the two, but perhaps there is.

Yaw also reported that the Cascading Select itemsets on
opendatakit.appspot.com are not working with the 1.4.7 rev 1053
release (and
they work but give odd 'null' resolutions on 1057). This is in a
related
area of this same code, so I will hopefully gain a better
understanding of
dynamic itemsets as I uncover the causes of that problem.


Mitch

On Mon, Nov 23, 2015 at 5:59 AM, Christopher Robert cro...@surveycto.com wrote:

So just to be clear, the "fix" we're testing here reverts the
entire
context within that populateDynamicChoices() function, to be the
field
instead of the parent -- so the effects are potentially much
wider than just
what current() returns. Potentially anything and everything that
depends on
context can change now, since the context will shift from parent
to field.

This means that we revert a change from 5 years ago and alter the
context from now on. Neither Meletis nor I feel terribly
confident that we
can entirely predict the consequences to backward compatibility.
We're
testing the things we can think to test, but honestly it's XML
users in the
broader ODK community who face the greatest risks here; SurveyCTO
users by
and large use XLSForm and thus the potential XPath syntax
consequences are
sharply limited.

Are you guys very sure that you want to take this path? I can tell
you now that if we make this change and then there is some outcry
after the
next ODK Collect release, SurveyCTO Collect will very likely
stick with the
new change (since we will have changed pyxform and we'll then
have too many
backward-compatibility concerns to then reverse the change).

Thanks,

Chris

On Mon, Nov 23, 2015 at 5:23 AM mel...@surveycto.com wrote:

Just for the history's sake, I found the time where the behavior
changed. It was at March 11, 2010:

https://bitbucket.org/m.sundt/javarosa/commits/b41875b2c23c9a21a868a5d2dc0ffc1f6b5b0b65#Lcore/src/org/javarosa/core/model/FormDef.javaT635

Take a look only in the changes of the "populateDynamicChoices"
method. The context reference used to be the "current" field,
but after the
changes above it started be the field's parent (by adding
getParent()).
Perhaps there was a reason for that change, but perhaps not, but
I think all
of us agree that this should be fixed, even after 5 and a half
years.

Meletis

On Monday, November 23, 2015 at 12:08:47 PM UTC+2, mel...@surveycto.com wrote:

Hi Mitch,

Please take a look at the code comment in line 31 here:

https://bitbucket.org/m.sundt/javarosa/src/bf17fb749b74e53915f086ca3827c3c6fe73fd07/core/src/org/javarosa/core/model/ItemsetBinding.java?at=default&fileviewer=file-view-default#ItemsetBinding.java-31

It clearly says: "ref of the control parent (group/formdef) of
itemset question"

So, for some reason, they wanted it to point to the parent. I
also
see that when the dynamic choices are populated, the evaluation
context is
wrapped around that parent (contextRef object), that's why
current() refers
to the parent group when evaluating itemsets nodesets.

I will run a few tests by changing the contextRef field value, I
just wanted to mention that in case you remember a reason for
that code
comment.

Meletis

On Friday, November 20, 2015 at 8:49:12 PM UTC+2, Mitch wrote:

Yes, I think it should be a one-line change; just don't know
where.

Mitch

(I do jar-updates over Christmas / New-Years. Tedious, but
doesn't require a whole lot of attention.)

On Fri, Nov 20, 2015 at 10:36 AM, mel...@surveycto.com wrote:

Hi Mitch,

Thank you for all the details, I guess I was missing some of
the
context of the original issue.

We will give it a shot to see if we can come up with a few
ideas
how to fix the evaluation content of the itemset nodeset. I
just hope it
will not be like last year with the "form processing logic"
in JR, otherwise
I believe that every Christmas will be a JR Christmas. :slight_smile:

Meletis

On Friday, November 20, 2015 at 7:29:14 PM UTC+2, Mitch wrote:

The form was attached in one of my earlier replies.

On Fri, Nov 20, 2015 at 9:27 AM, Mitch Sundt mitche...@gmail.com wrote:

And you see this in constraint expressions where you write:

. > 6

I.e., current()/. should refer to the field, and not the
group
containing the field, as it currently does when evaluating
expressions in
the itemset nodeset.

On Fri, Nov 20, 2015 at 9:26 AM, Mitch Sundt mitche...@gmail.com wrote:

The issue is that there is a discrepancy in the meaning of
'.' across the different contexts of: calculate, relevant,
, and
these itemset nodeset expressions.

In other usages, you need to reference this other field
with
current()/../filterfield ; but in the nodeset expression,
you would
reference it with current()/./filterfield

I.e., the notion of '.' is different when used within a
nodeset expression and when used within a constraint
expression or an
field.

See the attached NBiggestOfSet form. For the label, it
has:

    <select1 ref="/NBiggestOfSet/education/edu_level">
      <label>What is <output

value="current()/../referred_attendee_name"/>'s level of
education?
...

For a constraint, it has:
<bind

calculate="/NBiggestOfSet/attendee/name[position(..)=int(current()/../referred_attendee)]"

nodeset="/NBiggestOfSet/education/referred_attendee_name"
readonly="true()"
type="string"/>

These each use the current()/../ to reference the group (
/NBiggesetOfSet/education ).

But, within the nodeset definition (if this form used one),
you would need to use current()/./referred_attendee_name
to reference a
field in that group.

On Fri, Nov 20, 2015 at 8:22 AM, mel...@surveycto.com wrote:

Hi Mitch,

I think I'm missing something but I think that nothing is
broken and that the only issue is that the expression
current()/../filterfield in your example above should be
just changed to
current()/filterfield. If you make that change,
everything should work
because it will be resolved to:

/data/group1/filterfield

Why should a "../" be present there? We have used
current()/somefield successfully in one of our test forms
and it works fine.
And I think it would also work great in your example.
Wouldn't it? Or am I
totally out of context?

Meletis

On Friday, November 20, 2015 at 5:35:16 PM UTC+2, Mitch wrote:

Hi Meletis,

That's not the issue --

The issue is that if you use a filter condition in the
nodeset specification of a field "myfield",
within that nodeset
specification "current()/.." refers to the parent of
the group enclosing
the "myfield" field. It does not refer to the group
enclosing the "myfield"
field.

So if you have

a 4 fish b 14 fish ...

and if you have:

<itemset

nodeset="instance('food')/root/fooditem[filtercriteria=current()/../filterfield]">

        <value ref="code"/>
        <label ref="name"/>
</itemset>

This will currently be broken. Because
current()/../filterfield,
when evaluated within the itemset nodeset specification,
is
resolved to:

/data/filterfield

instead of

/data/group1/filterfield

On Fri, Nov 20, 2015 at 1:58 AM, mel...@surveycto.com wrote:

Well, I think there is a fundamental issue here
regarding
current() syntax. Please, let me remind you how JR
works with "current()":

  1. The expression "current()" looks like a function but
    it
    is not treated like one; it is not handled by the
    XPathFuncExpr class, so,
    if you try to use it alone you will get an error that
    the "function" is not
    supported. In other words, you cannot use it alone in
    an expression in order
    to refer to the "current" field.

  2. When an expression includes "current()", then it
    ALWAYS
    needs to be used in an XPath, actually, the expression
    needs to BE an XPath
    expression, so that it can be handled with the
    XPathPathExpr class. For that
    reason the expression NEEDS to contain at least one "/".

  3. Taking 1 and 2 into consideration, the ONLY way to
    refer to the "current" field (named "somefield") is to
    use "current()/." or
    "current()/somefield". So, I looks like that the
    original implementation was
    "okay" in the sense that the "current()" should only be
    used in XPaths.

If we wanted to have "current()" refer to the current
field, I believe we should introduce a new function
handler in XPathFuncExpr
class. Please let me know if this would work:

A. Whenever the "current()" expression is used inside an
XPath, then the current code would be used as it is
used right now
(maintaining backwards compatibility), so that it will
still be referring to
the enclosing group, not the field itself.
B. Whenever the "current()" expression is used inside a
non-XPath, then the new code would be used (in
XPathFuncExpr) and that would
be evaluated in a way that the current field would be
returned/referenced,
NOT the enclosing group.

That way, the following code could work, because it
wouldn't throw the "unrecognized function" error:

label .... ....

What do you think about that approach?

Meletis

On Friday, November 13, 2015 at 11:38:15 PM UTC+2, Martijn van de Rijdt wrote:

Hi,

I'd like to ask for a clarification about the current()
node in an XPath predicate. It seems JR considers this
to be the parent node
of the current question. (Thanks to Chris for sharing
this.)

E.g. in this question below, current() equates to
/data/group and not to /data/group/options. I would
have expected it to be
the latter.

....
    <label>label</label>

    <itemset

nodeset="instance('crop_list')/root/item[name =
current()/../crop]">

         ....

         ....

    </itemset>

Is this by design? Or is it a bug?

We ran into this by realizing that the seemingly more
intuitive predicate [name = current()/../../crop] is
not working.

See this form that illustrates the behaviour nicely. If
current() is used in a calculation (even if it makes
no sense to do so) this
extra step down is not taken, but in a predicate it
is. So it seems the
behaviour is not related to the current() function but
to how predicates are
handled in JR.

Thanks,
Martijn

--

--
Mitch Sundt
Software Engineer
University of Washington
mitche...@gmail.com

--

--
Mitch Sundt
Software Engineer
University of Washington
mitche...@gmail.com

--
Mitch Sundt
Software Engineer
University of Washington
mitche...@gmail.com

--
Mitch Sundt
Software Engineer
University of Washington
mitche...@gmail.com

--
You received this message because you are subscribed to the
Google Groups "ODK Developers" group.
To unsubscribe from this group and stop receiving emails from
it, send an email to
opendatakit-developers+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--
Mitch Sundt
Software Engineer
University of Washington
mitche...@gmail.com

--
You received this message because you are subscribed to the
Google
Groups "ODK Developers" group.
To unsubscribe from this group and stop receiving emails from it,
send an email to
opendatakit-developers+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google
Groups "ODK Developers" group.
To unsubscribe from this group and stop receiving emails from it,
send an email to
opendatakit-developers+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--

Mitch Sundt
Software Engineer
University of Washington
mitche...@gmail.com

--
Mitch Sundt
Software Engineer
University of Washington
mitche...@gmail.com

--
You received this message because you are subscribed to the Google
Groups "ODK Developers" group.
To unsubscribe from this group and stop receiving emails from it,
send
an email to opendatakit-developers+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--

You received this message because you are subscribed to a topic in
the
Google Groups "ODK Developers" group.
To unsubscribe from this topic, visit

https://groups.google.com/d/topic/opendatakit-developers/nN8HTA5pUIM/unsubscribe
.

To unsubscribe from this group and all its topics, send an email to
opendatakit-developers+unsubscribe@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--
Revolutionizing data collection since 2012.

Enketo | LinkedIn | GitHub | Twitter | Blog

--
You received this message because you are subscribed to the Google
Groups "ODK Developers" group.
To unsubscribe from this group and stop receiving emails from it, send
an email to opendatakit-developers+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--

Mitch Sundt
Software Engineer
University of Washington

mitche...@gmail.com

--
You received this message because you are subscribed to the Google
Groups
"ODK Developers" group.
To unsubscribe from this group and stop receiving emails from it, send
an
email to opendatakit-developers+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups
"ODK Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to opendatakit-developers+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups
"ODK Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to opendatakit-developers+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

I can't help thinking that we're overthinking this backwards-compatibility
issue.

Ideally we have old clients that have the bug and new clients that do not
have the bug. Isn't that the case with any other major bug fix? If an old
form stops working in the new version of ODK Collect, the survey owner will
either have to upload a new version of the form or the user will have to
downgrade ODK Collect. Surely this will be a problem for some users, but
for how many? A small fraction of one percent seems like a reasonable guess.

Maybe the backwards-compatibility should be addressed just by warning
users, as Yaw suggested way above.

Any 'side-effects' of fixing the bug, like with creating itemsets, can be
tested for, I presume, before deploying the fix.

One way to detect at least a significant portion of the current current()
users, is to do a regex search for any usage of the
'current()/path/to/node' where current()/ is not followed by '../'. That
usage is specific to this bug to refer to a sibling node and will not occur
any more after the bug is fixed (because a question cannot have children).

··· On Mon, Nov 30, 2015 at 11:43 AM, Christopher Robert wrote:

I wasn't going to suggest adding the different parameters to any of the
XLSForm templates -- so they wouldn't become cluttered with this stuff.
Really, it would just be an emergency option for somebody requiring
different compatibility for some component that was out of sync with their
other components. Ideally, nobody would actually have to use the settings
sheet to override any of these options.

To be specific, my proposal would be that everybody
support current_always_field=false today, and that this be how pyxform
today defaults. Ideally, this would include Enketo, as Collect and
SurveyCTO's Web Collect (and anything else using the ODK JavaRosa) already
support current_always_field=false as their current behavior. Then, we
figure out how to get current_always_field=true to work in JavaRosa and
phase that in, eventually changing the pyxform default to that once we
think that enough components support it.

We have thousands of devices out there using older Collect versions, even
well after their servers have been updated... so my guess is that it's some
of those users who will end up having to use the settings-sheet option to
force current_always_field=false after we've changed the default to true.

Does all of that make sense?

Chris

On Sun, Nov 29, 2015 at 6:48 PM Yaw Anokwa yanokwa@nafundi.com wrote:

All this sounds fine to me, but wanted to raise some potential issues.
Discrete parameters also means folks can pick and choose what behavior
they want and it's hard to move the entire community forward that way.
Also, once you get past two or three, you end up with a settings sheet
of lots of flags to toggle and that isn't a great user experience.

Yaw

Need ODK consultants? Nafundi provides form design, server setup,
in-field training, and software development for ODK. Go to
https://nafundi.com to get started.

On Sun, Nov 29, 2015 at 6:47 PM, Christopher Robert crobert@surveycto.com wrote:

Having a number of discrete parameters that are overridable on the
settings
sheet makes sense to me (like "current_references_parent" or perhaps
"current_always_field"). We could then add model parameters and change
the
pyxform default over time, but users could always override on their
settings
sheets as needed for specific compatibility.

For example, our SurveyCTO pyxform for 2.10 (we've given up on 2.01 now)
could default to current_always_field=false, which would mean that JR
would
be backward-compatible and pyxform would emit relative paths for the
choice_filter column assuming that current() references the parent. This
would break all choice_filters in Enketo, but our Enketo users could set
current_always_field=true in their settings sheets.

For that to work, though, we'd also have to add support for
current_always_field=true in JR. We could do that as Meletis has
proposed,
which may have other side effects -- we'll have to see (e.g., for
itemset
copying).

Alternatively, we could try to convince Martijn to support
current_always_field=false, which is likely trivial. If he would do
that,
then we could push JR's support for current_always_field=true to when we
could figure out how to do it safely. Also, it would mean producing a
version of Enketo that supports relative paths in choice filters in a
manner
compatible with ODK Collect. Ona, Kobo, and other Enketo users could
rejoice, able to produce forms that use current() (like those that
reference
same-repeat values) in their choice_filters. That boon to users has to
be
worth something, even if a principle is slightly violated (supporting
something outside the XPath spec).

Chris

On Tue, Nov 24, 2015 at 8:43 PM Alex Dorey dorey415@gmail.com wrote:

Hi, Just catching up on the discussion. Sorry if I'm a bit behind in my
understanding of things.

Regarding the pyxform changes--

  1. Instead of the date, could we have the configuration parameter
    describe
    the specific change?

So instead of:

revision="2015-11-24"

we specify something like:

current_references_parent=True (default to False with the next pyxform
version bump)

  1. Could we put this configuration into the xlsform's settings sheet? I
    think this would be easier to test and configure. We could also add a
    way to
    override the setting with an extra argument to the CLI or xls2json
    call.

On Tuesday, November 24, 2015 at 3:26:18 PM UTC-5, Mitch wrote:

I agree. Having just delved into the itemset code, there are a number
of
issues that would need to be cleaned up.

What I would propose is to leave JR / ODK Collect broken as-is. This
will be future work.

The python code will then be the focus of the tweaks.

(1) change:

json_survey =
xls2json.workbook_to_json(json.loads(request.POST['workbookJson']),
form_name=form_name, warnings=warningsList)

to add an optional revision argument:

json_survey =
xls2json.workbook_to_json(json.loads(request.POST['workbookJson']),
form_name=form_name, warnings=warningsList, revision='2015-11-24')

(2) change that implementation to:

(2a) store the revision string somewhere in the returned json object.

(2b) expand ${fieldname} with the new relative path expansion only if
the
passed-in revision is >= '2015-11-24' ; otherwise, use the absolute
path
expansion.

(3) change:

create_survey_element_from_dict()

to emit the revision value as an attribute on the element.

===============
The JR itemset bug will be fixed at a later date.

When that is fixed, we can introduce a new revision string for that
behavior change.

i.e., once we have the revision string moving through the system, we
can
begin to support non-backward-compatible changes to behaviors in
XLSForm and
JR form processing.

For the time being, the ODK XLSForm converter will not supply a
revision
argument, and not make use of this pyxform enhancement. Once the JR
fix is
done, we will then support that new revision and the current behavior,
defaulting to the new behavior, but supporting users in generating
the old
behavior.

Mitch

On Tue, Nov 24, 2015 at 8:44 AM, Martijn van de Rijdt < mar...@enketo.org> wrote:

I appreciate the creative solution. It does sound like it will
become a
bit of a headache to manage and explain to users though. Whether
it's worth
it depends on how many active forms are out there that rely on the
bug. We
should get Ona's and KoBo's input on this too, I think.

From Enketo's side, relative paths were never supported in
choice-filters (to my embarrassment and surprise - which led to my
question). As a result, there is no significant issue with backwards
compatibility in Enketo once it is fixed in JR. Not sure how
meaningful this
is to ODK Collect, but no Enketo user has ever reported this lack of
support. I think to date it's been too hard to craft such
choice-filters.
Because of the way Enketo is integrated into KoBo, Ona, Formhub, I'd
be
surprised if there were a significant amount of those users using
relative
choice-filters.

I do indeed feel strongly about following the XForms 2.0 spec here. I
think relative path support is going to be very useful in the
future, and I
agree that we need to look at the bigger picture and do this
correctly, so
it won't catch up with us.

On Tue, Nov 24, 2015 at 4:24 AM, Christopher Robert cro...@surveycto.com wrote:

Okay, well, this is turning into a bigger and bigger project and we
may
well miss the window for our 2.01 maintenance release (and/or make
the
changes so extensive and scary that we cannot in good conscience
include
them in a maintenance release).

However, your idea for making JR sensitive to the revision value
emitted by pyxform opens a new possibility: we could release the
pyxform
change with 2.01 so that users can begin to benefit from the
relative paths
(relying on today's current() behavior), then assume that when JR
is "fixed"
it will only be fixed for XML emitted by newer pyxforms that would
have
updated their paths to account for the fix (using your
revision-value idea).
The primary drawback to this approach is that we would basically
break all
cascading-select filter expressions for SurveyCTO+Enketo users
(since
Martijn clearly wants to implement the "fixed" version of current()
rather
than the JavaRosa-compatible version of current()).

Meletis also found another issue relating to the evaluation context
in
jr:choice-name() (
https://github.com/opendatakit/opendatakit/issues/1180),
so we have that to contend with as well. If we can't safely resolve
the
context for jr:choice-name(), then that would present another major
drawback
and further lessen our enthusiasm for this entire adventure.

Best,

Chris

On Mon, Nov 23, 2015 at 5:45 PM Mitch Sundt mitche...@gmail.com wrote:

And, if it isn't clear, to preserve backward-compatible operations,
XLSForm should not expand ${fieldname} with a relative path unless
the
revision value is the newer one; if the revision is not specified,
it would
emit absolute paths like it currently does.

I also switched to use a date value for the revision value
("2015-11-23") which allows simple lexical comparisons.

On Mon, Nov 23, 2015 at 1:03 PM, Mitch Sundt mitche...@gmail.com wrote:

w.r.t. backward compatibility

I too am concerned.

A proposed solution:

  1. Add a command line argument or other mechanism to request that
    XLSForm inject a 'revision="2015-11-23"' attribute into the
    element.
    e.g.,

This is used to alter the behavior of the XForms engine to
accommodate backward-compatible bug behaviors.

We can't identify the bugs (e.g, 'legacy="itemset-parent"'), since
that would break older forms run on the new ODK Collect releases.
We have to advance the revision code as bugs are fixed.

I think an integer value is fine for this.

I don't anticipate making a lot of revisions, and I think the
revisions would be sequentially applied (you can't
pick-and-choose what bugs
you include).

In the future, this would allow us to re-do the date-time
behavior.

  1. Modify the XLSForm tools (e.g., the online form generator) to
    enable the user to select what revision they want to emit (much
    like we did
    last year with the JR form eval choice setting within ODK
    Collect). We can
    default to emitting the latest revision if the user does not make
    a choice.

  2. Augment JR to look for this revision attribute and
    store/restore
    it in the FormDef binary object.

Based upon this value, alter the behavior of the JR itemset
evaluation to fix the path bug.

  1. Modify ODK Validate to scan for syntax that may be impacted by
    a
    revision (to advise whether to change something). Not sure how
    complex that
    might be. May not be always possible?

w.r.t. impacts of change

Thank you Meletis for researching when this change occurred.

I will have to ponder what effect the parent/node change would
have
on itemset copying. Hadn't thought that there was any
relationship between
the two, but perhaps there is.

Yaw also reported that the Cascading Select itemsets on
opendatakit.appspot.com are not working with the 1.4.7 rev 1053
release (and
they work but give odd 'null' resolutions on 1057). This is in a
related
area of this same code, so I will hopefully gain a better
understanding of
dynamic itemsets as I uncover the causes of that problem.


Mitch

On Mon, Nov 23, 2015 at 5:59 AM, Christopher Robert cro...@surveycto.com wrote:

So just to be clear, the "fix" we're testing here reverts the
entire
context within that populateDynamicChoices() function, to be the
field
instead of the parent -- so the effects are potentially much
wider than just
what current() returns. Potentially anything and everything that
depends on
context can change now, since the context will shift from parent
to field.

This means that we revert a change from 5 years ago and alter the
context from now on. Neither Meletis nor I feel terribly
confident that we
can entirely predict the consequences to backward compatibility.
We're
testing the things we can think to test, but honestly it's XML
users in the
broader ODK community who face the greatest risks here;
SurveyCTO users by
and large use XLSForm and thus the potential XPath syntax
consequences are
sharply limited.

Are you guys very sure that you want to take this path? I can
tell
you now that if we make this change and then there is some
outcry after the
next ODK Collect release, SurveyCTO Collect will very likely
stick with the
new change (since we will have changed pyxform and we'll then
have too many
backward-compatibility concerns to then reverse the change).

Thanks,

Chris

On Mon, Nov 23, 2015 at 5:23 AM mel...@surveycto.com wrote:

Just for the history's sake, I found the time where the behavior
changed. It was at March 11, 2010:

https://bitbucket.org/m.sundt/javarosa/commits/b41875b2c23c9a21a868a5d2dc0ffc1f6b5b0b65#Lcore/src/org/javarosa/core/model/FormDef.javaT635

Take a look only in the changes of the "populateDynamicChoices"
method. The context reference used to be the "current" field,
but after the
changes above it started be the field's parent (by adding
getParent()).
Perhaps there was a reason for that change, but perhaps not,
but I think all
of us agree that this should be fixed, even after 5 and a half
years.

Meletis

On Monday, November 23, 2015 at 12:08:47 PM UTC+2, mel...@surveycto.com wrote:

Hi Mitch,

Please take a look at the code comment in line 31 here:

https://bitbucket.org/m.sundt/javarosa/src/bf17fb749b74e53915f086ca3827c3c6fe73fd07/core/src/org/javarosa/core/model/ItemsetBinding.java?at=default&fileviewer=file-view-default#ItemsetBinding.java-31

It clearly says: "ref of the control parent (group/formdef) of
itemset question"

So, for some reason, they wanted it to point to the parent. I
also
see that when the dynamic choices are populated, the
evaluation context is
wrapped around that parent (contextRef object), that's why
current() refers
to the parent group when evaluating itemsets nodesets.

I will run a few tests by changing the contextRef field value,
I
just wanted to mention that in case you remember a reason for
that code
comment.

Meletis

On Friday, November 20, 2015 at 8:49:12 PM UTC+2, Mitch wrote:

Yes, I think it should be a one-line change; just don't know
where.

Mitch

(I do jar-updates over Christmas / New-Years. Tedious, but
doesn't require a whole lot of attention.)

On Fri, Nov 20, 2015 at 10:36 AM, mel...@surveycto.com wrote:

Hi Mitch,

Thank you for all the details, I guess I was missing some of
the
context of the original issue.

We will give it a shot to see if we can come up with a few
ideas
how to fix the evaluation content of the itemset nodeset. I
just hope it
will not be like last year with the "form processing logic"
in JR, otherwise
I believe that every Christmas will be a JR Christmas. :slight_smile:

Meletis

On Friday, November 20, 2015 at 7:29:14 PM UTC+2, Mitch wrote:

The form was attached in one of my earlier replies.

On Fri, Nov 20, 2015 at 9:27 AM, Mitch Sundt mitche...@gmail.com wrote:

And you see this in constraint expressions where you write:

. > 6

I.e., current()/. should refer to the field, and not the
group
containing the field, as it currently does when evaluating
expressions in
the itemset nodeset.

On Fri, Nov 20, 2015 at 9:26 AM, Mitch Sundt mitche...@gmail.com wrote:

The issue is that there is a discrepancy in the meaning of
'.' across the different contexts of: calculate,
relevant, , and
these itemset nodeset expressions.

In other usages, you need to reference this other field
with
current()/../filterfield ; but in the nodeset expression,
you would
reference it with current()/./filterfield

I.e., the notion of '.' is different when used within a
nodeset expression and when used within a constraint
expression or an
field.

See the attached NBiggestOfSet form. For the label, it
has:

    <select1 ref="/NBiggestOfSet/education/edu_level">
      <label>What is <output

value="current()/../referred_attendee_name"/>'s level of
education?
...

For a constraint, it has:
<bind

calculate="/NBiggestOfSet/attendee/name[position(..)=int(current()/../referred_attendee)]"

nodeset="/NBiggestOfSet/education/referred_attendee_name"
readonly="true()"
type="string"/>

These each use the current()/../ to reference the group (
/NBiggesetOfSet/education ).

But, within the nodeset definition (if this form used
one),
you would need to use current()/./referred_attendee_name
to reference a
field in that group.

On Fri, Nov 20, 2015 at 8:22 AM, mel...@surveycto.com wrote:

Hi Mitch,

I think I'm missing something but I think that nothing is
broken and that the only issue is that the expression
current()/../filterfield in your example above should be
just changed to
current()/filterfield. If you make that change,
everything should work
because it will be resolved to:

/data/group1/filterfield

Why should a "../" be present there? We have used
current()/somefield successfully in one of our test
forms and it works fine.
And I think it would also work great in your example.
Wouldn't it? Or am I
totally out of context?

Meletis

On Friday, November 20, 2015 at 5:35:16 PM UTC+2, Mitch wrote:

Hi Meletis,

That's not the issue --

The issue is that if you use a filter condition in the
nodeset specification of a field "myfield",
within that nodeset
specification "current()/.." refers to the parent of
the group enclosing
the "myfield" field. It does not refer to the group
enclosing the "myfield"
field.

So if you have

a 4 fish b 14 fish ...

and if you have:

<itemset

nodeset="instance('food')/root/fooditem[filtercriteria=current()/../filterfield]">

        <value ref="code"/>
        <label ref="name"/>
</itemset>

This will currently be broken. Because
current()/../filterfield,
when evaluated within the itemset nodeset
specification, is
resolved to:

/data/filterfield

instead of

/data/group1/filterfield

On Fri, Nov 20, 2015 at 1:58 AM, mel...@surveycto.com wrote:

Well, I think there is a fundamental issue here
regarding
current() syntax. Please, let me remind you how JR
works with "current()":

  1. The expression "current()" looks like a function
    but it
    is not treated like one; it is not handled by the
    XPathFuncExpr class, so,
    if you try to use it alone you will get an error that
    the "function" is not
    supported. In other words, you cannot use it alone in
    an expression in order
    to refer to the "current" field.

  2. When an expression includes "current()", then it
    ALWAYS
    needs to be used in an XPath, actually, the expression
    needs to BE an XPath
    expression, so that it can be handled with the
    XPathPathExpr class. For that
    reason the expression NEEDS to contain at least one
    "/".

  3. Taking 1 and 2 into consideration, the ONLY way to
    refer to the "current" field (named "somefield") is to
    use "current()/." or
    "current()/somefield". So, I looks like that the
    original implementation was
    "okay" in the sense that the "current()" should only
    be used in XPaths.

If we wanted to have "current()" refer to the current
field, I believe we should introduce a new function
handler in XPathFuncExpr
class. Please let me know if this would work:

A. Whenever the "current()" expression is used inside
an
XPath, then the current code would be used as it is
used right now
(maintaining backwards compatibility), so that it will
still be referring to
the enclosing group, not the field itself.
B. Whenever the "current()" expression is used inside a
non-XPath, then the new code would be used (in
XPathFuncExpr) and that would
be evaluated in a way that the current field would be
returned/referenced,
NOT the enclosing group.

That way, the following code could work, because it
wouldn't throw the "unrecognized function" error:

label
....
    <label>label</label>

    <itemset

nodeset="instance('crop_list')/root/item[name =
current()/../crop]">

         ....

         ....

    </itemset>

Is this by design? Or is it a bug?

We ran into this by realizing that the seemingly more
intuitive predicate [name = current()/../../crop] is
not working.

See this form that illustrates the behaviour nicely.
If
current() is used in a calculation (even if it makes
no sense to do so) this
extra step down is not taken, but in a predicate it
is. So it seems the
behaviour is not related to the current() function
but to how predicates are
handled in JR.

Thanks,
Martijn

--

--
Mitch Sundt
Software Engineer
University of Washington
mitche...@gmail.com

--

--
Mitch Sundt
Software Engineer
University of Washington
mitche...@gmail.com

--
Mitch Sundt
Software Engineer
University of Washington
mitche...@gmail.com

--
Mitch Sundt
Software Engineer
University of Washington
mitche...@gmail.com

--
You received this message because you are subscribed to the
Google Groups "ODK Developers" group.
To unsubscribe from this group and stop receiving emails from
it, send an email to
opendatakit-developers+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--
Mitch Sundt
Software Engineer
University of Washington
mitche...@gmail.com

--
You received this message because you are subscribed to the
Google
Groups "ODK Developers" group.
To unsubscribe from this group and stop receiving emails from
it,
send an email to
opendatakit-developers+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the
Google
Groups "ODK Developers" group.
To unsubscribe from this group and stop receiving emails from it,
send an email to
opendatakit-developers+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--

Mitch Sundt
Software Engineer
University of Washington
mitche...@gmail.com

--
Mitch Sundt
Software Engineer
University of Washington
mitche...@gmail.com

--
You received this message because you are subscribed to the Google
Groups "ODK Developers" group.
To unsubscribe from this group and stop receiving emails from it,
send
an email to opendatakit-developers+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--

You received this message because you are subscribed to a topic in
the
Google Groups "ODK Developers" group.
To unsubscribe from this topic, visit

https://groups.google.com/d/topic/opendatakit-developers/nN8HTA5pUIM/unsubscribe
.

To unsubscribe from this group and all its topics, send an email to
opendatakit-developers+unsubscribe@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--
Revolutionizing data collection since 2012.

Enketo | LinkedIn | GitHub | Twitter | Blog

--
You received this message because you are subscribed to the Google
Groups "ODK Developers" group.
To unsubscribe from this group and stop receiving emails from it,
send
an email to opendatakit-developers+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--

Mitch Sundt
Software Engineer
University of Washington

mitche...@gmail.com

--
You received this message because you are subscribed to the Google
Groups
"ODK Developers" group.
To unsubscribe from this group and stop receiving emails from it, send
an
email to opendatakit-developers+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google
Groups
"ODK Developers" group.
To unsubscribe from this group and stop receiving emails from it, send
an
email to opendatakit-developers+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups
"ODK Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to opendatakit-developers+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to a topic in the
Google Groups "ODK Developers" group.
To unsubscribe from this topic, visit
https://groups.google.com/d/topic/opendatakit-developers/nN8HTA5pUIM/unsubscribe
.
To unsubscribe from this group and all its topics, send an email to
opendatakit-developers+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--

Revolutionizing data collection since 2012.

Enketo https://enketo.org/ | LinkedIn
http://www.linkedin.com/company/enketo-llc | GitHub
https://github.com/enketo | Twitter https://twitter.com/enketo
| Blog http://blog.enketo.org/

We have individual users with 500+ devices in the field. Updating devices
-- or even keeping track of which devices are on which versions -- is a
major undertaking. In total, we're talking many thousands of devices spread
across many countries. The problem is not a small one, and our commitment
to backward compatibility is one that our users not only appreciate but
expect.

ODK/SurveyCTO Collect continues to use older versions of forms to view,
edit, or finalize submissions begun under those older versions, even when
newer versions exist. So the backward-compatibility challenge is a
formidable one. If you change the way JavaRosa interprets expressions in
the kind of blanket way that you suggest, then users will be faced with the
impossible choice of running software that fails to function properly with
existing, in-progress, or old submissions, or running software that fails
to function properly with newer submissions. Multiply the difficulty of the
choice and the potential confusion of XPath parsing errors times
however-many devices there are in the field.

No, this is a legitimately difficult situation that requires a tremendous
amount of care. Changing how XPath expressions are interpreted is a big
deal for the Collect world, though I can see why it would be considerably
less so in the Enketo world.

Now, I will admit that there are bound to be very few users who already use
current() in choice filters. Mitch suggested the approach a few times in
the public forum, but even I failed to get it to work the first several
times I tried to follow his advice. So chances are that not many projects
are using current() today. (I know of 1-2, maybe there are 8-10 total.
That's just a guess.)

Enketo aside, we could -- today! -- allow people to reference in-context
repeated values by exploiting the current JR support for current(). For
some users/projects, there would be an immediate benefit. But obviously
that relies on "the bug" and so it would expand the potential need for
backward compatibility. And it would break choice-filter compatibility
between Collect and Enketo (say, for SurveyCTO users at least).

It's easy to say "just fix the bug and then this problem goes away," but
the fact is that we don't quite know how to fix it without breaking other
things; we have a candidate for a fix, but it's seriously scary. It may
take time. And meanwhile, we have already gone multiple years with users
waiting to be able to reference within-context repeated values in choice
filters. If a safe fix drags on for another year or two -- either because
it's just too hard or because nobody quite has the time -- is that just the
way it goes? I don't know. It's not obvious to me that breaking
compatibility with Enketo and relying on current behavior isn't the
welfare-maximizing choice.

In the end, I believe that Mitch's suggestion re: compatibility is maybe
necessary (storing in the XML some indication of how current() in choice
filters should be interpreted). And if we totally screwed up the fix and
broke a bunch of use cases, at least it might offer some escape for
affected users (if we allowed an override on the settings sheet).

So no, I don't agree that it's as simple as "fix the bug and make people
upgrade."

Chris

··· On Tue, Dec 1, 2015 at 5:11 PM Martijn van de Rijdt wrote:

I can't help thinking that we're overthinking this backwards-compatibility
issue.

Ideally we have old clients that have the bug and new clients that do not
have the bug. Isn't that the case with any other major bug fix? If an old
form stops working in the new version of ODK Collect, the survey owner will
either have to upload a new version of the form or the user will have to
downgrade ODK Collect. Surely this will be a problem for some users, but
for how many? A small fraction of one percent seems like a reasonable guess.

Maybe the backwards-compatibility should be addressed just by warning
users, as Yaw suggested way above.

Any 'side-effects' of fixing the bug, like with creating itemsets, can be
tested for, I presume, before deploying the fix.

One way to detect at least a significant portion of the current current()
users, is to do a regex search for any usage of the
'current()/path/to/node' where current()/ is not followed by '../'. That
usage is specific to this bug to refer to a sibling node and will not occur
any more after the bug is fixed (because a question cannot have children).

On Mon, Nov 30, 2015 at 11:43 AM, Christopher Robert < crobert@surveycto.com> wrote:

I wasn't going to suggest adding the different parameters to any of the
XLSForm templates -- so they wouldn't become cluttered with this stuff.
Really, it would just be an emergency option for somebody requiring
different compatibility for some component that was out of sync with their
other components. Ideally, nobody would actually have to use the settings
sheet to override any of these options.

To be specific, my proposal would be that everybody
support current_always_field=false today, and that this be how pyxform
today defaults. Ideally, this would include Enketo, as Collect and
SurveyCTO's Web Collect (and anything else using the ODK JavaRosa) already
support current_always_field=false as their current behavior. Then, we
figure out how to get current_always_field=true to work in JavaRosa and
phase that in, eventually changing the pyxform default to that once we
think that enough components support it.

We have thousands of devices out there using older Collect versions, even
well after their servers have been updated... so my guess is that it's some
of those users who will end up having to use the settings-sheet option to
force current_always_field=false after we've changed the default to true.

Does all of that make sense?

Chris

On Sun, Nov 29, 2015 at 6:48 PM Yaw Anokwa yanokwa@nafundi.com wrote:

All this sounds fine to me, but wanted to raise some potential issues.
Discrete parameters also means folks can pick and choose what behavior
they want and it's hard to move the entire community forward that way.
Also, once you get past two or three, you end up with a settings sheet
of lots of flags to toggle and that isn't a great user experience.

Yaw

Need ODK consultants? Nafundi provides form design, server setup,
in-field training, and software development for ODK. Go to
https://nafundi.com to get started.

On Sun, Nov 29, 2015 at 6:47 PM, Christopher Robert crobert@surveycto.com wrote:

Having a number of discrete parameters that are overridable on the
settings
sheet makes sense to me (like "current_references_parent" or perhaps
"current_always_field"). We could then add model parameters and change
the
pyxform default over time, but users could always override on their
settings
sheets as needed for specific compatibility.

For example, our SurveyCTO pyxform for 2.10 (we've given up on 2.01
now)
could default to current_always_field=false, which would mean that JR
would
be backward-compatible and pyxform would emit relative paths for the
choice_filter column assuming that current() references the parent.
This
would break all choice_filters in Enketo, but our Enketo users could
set
current_always_field=true in their settings sheets.

For that to work, though, we'd also have to add support for
current_always_field=true in JR. We could do that as Meletis has
proposed,
which may have other side effects -- we'll have to see (e.g., for
itemset
copying).

Alternatively, we could try to convince Martijn to support
current_always_field=false, which is likely trivial. If he would do
that,
then we could push JR's support for current_always_field=true to when
we
could figure out how to do it safely. Also, it would mean producing a
version of Enketo that supports relative paths in choice filters in a
manner
compatible with ODK Collect. Ona, Kobo, and other Enketo users could
rejoice, able to produce forms that use current() (like those that
reference
same-repeat values) in their choice_filters. That boon to users has to
be
worth something, even if a principle is slightly violated (supporting
something outside the XPath spec).

Chris

On Tue, Nov 24, 2015 at 8:43 PM Alex Dorey dorey415@gmail.com wrote:

Hi, Just catching up on the discussion. Sorry if I'm a bit behind in
my
understanding of things.

Regarding the pyxform changes--

  1. Instead of the date, could we have the configuration parameter
    describe
    the specific change?

So instead of:

revision="2015-11-24"

we specify something like:

current_references_parent=True (default to False with the next
pyxform
version bump)

  1. Could we put this configuration into the xlsform's settings sheet?
    I
    think this would be easier to test and configure. We could also add a
    way to
    override the setting with an extra argument to the CLI or xls2json
    call.

On Tuesday, November 24, 2015 at 3:26:18 PM UTC-5, Mitch wrote:

I agree. Having just delved into the itemset code, there are a
number of
issues that would need to be cleaned up.

What I would propose is to leave JR / ODK Collect broken as-is. This
will be future work.

The python code will then be the focus of the tweaks.

(1) change:

json_survey =
xls2json.workbook_to_json(json.loads(request.POST['workbookJson']),
form_name=form_name, warnings=warningsList)

to add an optional revision argument:

json_survey =
xls2json.workbook_to_json(json.loads(request.POST['workbookJson']),
form_name=form_name, warnings=warningsList, revision='2015-11-24')

(2) change that implementation to:

(2a) store the revision string somewhere in the returned json object.

(2b) expand ${fieldname} with the new relative path expansion only
if the
passed-in revision is >= '2015-11-24' ; otherwise, use the absolute
path
expansion.

(3) change:

create_survey_element_from_dict()

to emit the revision value as an attribute on the element.

===============
The JR itemset bug will be fixed at a later date.

When that is fixed, we can introduce a new revision string for that
behavior change.

i.e., once we have the revision string moving through the system, we
can
begin to support non-backward-compatible changes to behaviors in
XLSForm and
JR form processing.

For the time being, the ODK XLSForm converter will not supply a
revision
argument, and not make use of this pyxform enhancement. Once the JR
fix is
done, we will then support that new revision and the current
behavior,
defaulting to the new behavior, but supporting users in generating
the old
behavior.

Mitch

On Tue, Nov 24, 2015 at 8:44 AM, Martijn van de Rijdt < mar...@enketo.org> wrote:

I appreciate the creative solution. It does sound like it will
become a
bit of a headache to manage and explain to users though. Whether
it's worth
it depends on how many active forms are out there that rely on the
bug. We
should get Ona's and KoBo's input on this too, I think.

From Enketo's side, relative paths were never supported in
choice-filters (to my embarrassment and surprise - which led to my
question). As a result, there is no significant issue with backwards
compatibility in Enketo once it is fixed in JR. Not sure how
meaningful this
is to ODK Collect, but no Enketo user has ever reported this lack of
support. I think to date it's been too hard to craft such
choice-filters.
Because of the way Enketo is integrated into KoBo, Ona, Formhub,
I'd be
surprised if there were a significant amount of those users using
relative
choice-filters.

I do indeed feel strongly about following the XForms 2.0 spec here.
I
think relative path support is going to be very useful in the
future, and I
agree that we need to look at the bigger picture and do this
correctly, so
it won't catch up with us.

On Tue, Nov 24, 2015 at 4:24 AM, Christopher Robert cro...@surveycto.com wrote:

Okay, well, this is turning into a bigger and bigger project and
we may
well miss the window for our 2.01 maintenance release (and/or make
the
changes so extensive and scary that we cannot in good conscience
include
them in a maintenance release).

However, your idea for making JR sensitive to the revision value
emitted by pyxform opens a new possibility: we could release the
pyxform
change with 2.01 so that users can begin to benefit from the
relative paths
(relying on today's current() behavior), then assume that when JR
is "fixed"
it will only be fixed for XML emitted by newer pyxforms that would
have
updated their paths to account for the fix (using your
revision-value idea).
The primary drawback to this approach is that we would basically
break all
cascading-select filter expressions for SurveyCTO+Enketo users
(since
Martijn clearly wants to implement the "fixed" version of
current() rather
than the JavaRosa-compatible version of current()).

Meletis also found another issue relating to the evaluation
context in
jr:choice-name() (
https://github.com/opendatakit/opendatakit/issues/1180),
so we have that to contend with as well. If we can't safely
resolve the
context for jr:choice-name(), then that would present another
major drawback
and further lessen our enthusiasm for this entire adventure.

Best,

Chris

On Mon, Nov 23, 2015 at 5:45 PM Mitch Sundt mitche...@gmail.com wrote:

And, if it isn't clear, to preserve backward-compatible
operations,
XLSForm should not expand ${fieldname} with a relative path
unless the
revision value is the newer one; if the revision is not
specified, it would
emit absolute paths like it currently does.

I also switched to use a date value for the revision value
("2015-11-23") which allows simple lexical comparisons.

On Mon, Nov 23, 2015 at 1:03 PM, Mitch Sundt <mitche...@gmail.com wrote:

w.r.t. backward compatibility

I too am concerned.

A proposed solution:

  1. Add a command line argument or other mechanism to request that
    XLSForm inject a 'revision="2015-11-23"' attribute into the
    element.
    e.g.,

This is used to alter the behavior of the XForms engine to
accommodate backward-compatible bug behaviors.

We can't identify the bugs (e.g, 'legacy="itemset-parent"'),
since
that would break older forms run on the new ODK Collect releases.
We have to advance the revision code as bugs are fixed.

I think an integer value is fine for this.

I don't anticipate making a lot of revisions, and I think the
revisions would be sequentially applied (you can't
pick-and-choose what bugs
you include).

In the future, this would allow us to re-do the date-time
behavior.

  1. Modify the XLSForm tools (e.g., the online form generator) to
    enable the user to select what revision they want to emit (much
    like we did
    last year with the JR form eval choice setting within ODK
    Collect). We can
    default to emitting the latest revision if the user does not
    make a choice.

  2. Augment JR to look for this revision attribute and
    store/restore
    it in the FormDef binary object.

Based upon this value, alter the behavior of the JR itemset
evaluation to fix the path bug.

  1. Modify ODK Validate to scan for syntax that may be impacted
    by a
    revision (to advise whether to change something). Not sure how
    complex that
    might be. May not be always possible?

w.r.t. impacts of change

Thank you Meletis for researching when this change occurred.

I will have to ponder what effect the parent/node change would
have
on itemset copying. Hadn't thought that there was any
relationship between
the two, but perhaps there is.

Yaw also reported that the Cascading Select itemsets on
opendatakit.appspot.com are not working with the 1.4.7 rev 1053
release (and
they work but give odd 'null' resolutions on 1057). This is in a
related
area of this same code, so I will hopefully gain a better
understanding of
dynamic itemsets as I uncover the causes of that problem.


Mitch

On Mon, Nov 23, 2015 at 5:59 AM, Christopher Robert cro...@surveycto.com wrote:

So just to be clear, the "fix" we're testing here reverts the
entire
context within that populateDynamicChoices() function, to be
the field
instead of the parent -- so the effects are potentially much
wider than just
what current() returns. Potentially anything and everything
that depends on
context can change now, since the context will shift from
parent to field.

This means that we revert a change from 5 years ago and alter
the
context from now on. Neither Meletis nor I feel terribly
confident that we
can entirely predict the consequences to backward
compatibility. We're
testing the things we can think to test, but honestly it's XML
users in the
broader ODK community who face the greatest risks here;
SurveyCTO users by
and large use XLSForm and thus the potential XPath syntax
consequences are
sharply limited.

Are you guys very sure that you want to take this path? I can
tell
you now that if we make this change and then there is some
outcry after the
next ODK Collect release, SurveyCTO Collect will very likely
stick with the
new change (since we will have changed pyxform and we'll then
have too many
backward-compatibility concerns to then reverse the change).

Thanks,

Chris

On Mon, Nov 23, 2015 at 5:23 AM mel...@surveycto.com wrote:

Just for the history's sake, I found the time where the
behavior
changed. It was at March 11, 2010:

https://bitbucket.org/m.sundt/javarosa/commits/b41875b2c23c9a21a868a5d2dc0ffc1f6b5b0b65#Lcore/src/org/javarosa/core/model/FormDef.javaT635

Take a look only in the changes of the "populateDynamicChoices"
method. The context reference used to be the "current" field,
but after the
changes above it started be the field's parent (by adding
getParent()).
Perhaps there was a reason for that change, but perhaps not,
but I think all
of us agree that this should be fixed, even after 5 and a half
years.

Meletis

On Monday, November 23, 2015 at 12:08:47 PM UTC+2, mel...@surveycto.com wrote:

Hi Mitch,

Please take a look at the code comment in line 31 here:

https://bitbucket.org/m.sundt/javarosa/src/bf17fb749b74e53915f086ca3827c3c6fe73fd07/core/src/org/javarosa/core/model/ItemsetBinding.java?at=default&fileviewer=file-view-default#ItemsetBinding.java-31

It clearly says: "ref of the control parent (group/formdef) of
itemset question"

So, for some reason, they wanted it to point to the parent. I
also
see that when the dynamic choices are populated, the
evaluation context is
wrapped around that parent (contextRef object), that's why
current() refers
to the parent group when evaluating itemsets nodesets.

I will run a few tests by changing the contextRef field
value, I
just wanted to mention that in case you remember a reason for
that code
comment.

Meletis

On Friday, November 20, 2015 at 8:49:12 PM UTC+2, Mitch wrote:

Yes, I think it should be a one-line change; just don't know
where.

Mitch

(I do jar-updates over Christmas / New-Years. Tedious, but
doesn't require a whole lot of attention.)

On Fri, Nov 20, 2015 at 10:36 AM, mel...@surveycto.com wrote:

Hi Mitch,

Thank you for all the details, I guess I was missing some
of the
context of the original issue.

We will give it a shot to see if we can come up with a few
ideas
how to fix the evaluation content of the itemset nodeset. I
just hope it
will not be like last year with the "form processing logic"
in JR, otherwise
I believe that every Christmas will be a JR Christmas. :slight_smile:

Meletis

On Friday, November 20, 2015 at 7:29:14 PM UTC+2, Mitch wrote:

The form was attached in one of my earlier replies.

On Fri, Nov 20, 2015 at 9:27 AM, Mitch Sundt mitche...@gmail.com wrote:

And you see this in constraint expressions where you
write:

. > 6

I.e., current()/. should refer to the field, and not the
group
containing the field, as it currently does when
evaluating expressions in
the itemset nodeset.

On Fri, Nov 20, 2015 at 9:26 AM, Mitch Sundt mitche...@gmail.com wrote:

The issue is that there is a discrepancy in the meaning
of
'.' across the different contexts of: calculate,
relevant, , and
these itemset nodeset expressions.

In other usages, you need to reference this other field
with
current()/../filterfield ; but in the nodeset
expression, you would
reference it with current()/./filterfield

I.e., the notion of '.' is different when used within a
nodeset expression and when used within a constraint
expression or an
field.

See the attached NBiggestOfSet form. For the label, it
has:

    <select1

ref="/NBiggestOfSet/education/edu_level">

      <label>What is <output

value="current()/../referred_attendee_name"/>'s level of
education?
...

For a constraint, it has:
<bind

calculate="/NBiggestOfSet/attendee/name[position(..)=int(current()/../referred_attendee)]"

nodeset="/NBiggestOfSet/education/referred_attendee_name" readonly="true()"

type="string"/>

These each use the current()/../ to reference the group (
/NBiggesetOfSet/education ).

But, within the nodeset definition (if this form used
one),
you would need to use
current()/./referred_attendee_name to reference a
field in that group.

On Fri, Nov 20, 2015 at 8:22 AM, mel...@surveycto.com wrote:

Hi Mitch,

I think I'm missing something but I think that nothing
is
broken and that the only issue is that the expression
current()/../filterfield in your example above should
be just changed to
current()/filterfield. If you make that change,
everything should work
because it will be resolved to:

/data/group1/filterfield

Why should a "../" be present there? We have used
current()/somefield successfully in one of our test
forms and it works fine.
And I think it would also work great in your example.
Wouldn't it? Or am I
totally out of context?

Meletis

On Friday, November 20, 2015 at 5:35:16 PM UTC+2, Mitch wrote:

Hi Meletis,

That's not the issue --

The issue is that if you use a filter condition in the
nodeset specification of a field "myfield",
within that nodeset
specification "current()/.." refers to the parent of
the group enclosing
the "myfield" field. It does not refer to the group
enclosing the "myfield"
field.

So if you have

a 4 fish b 14 fish ...

and if you have:

<itemset

nodeset="instance('food')/root/fooditem[filtercriteria=current()/../filterfield]">

        <value ref="code"/>
        <label ref="name"/>
</itemset>

This will currently be broken. Because
current()/../filterfield,
when evaluated within the itemset nodeset
specification, is
resolved to:

/data/filterfield

instead of

/data/group1/filterfield

On Fri, Nov 20, 2015 at 1:58 AM, <mel...@surveycto.com wrote:

Well, I think there is a fundamental issue here
regarding
current() syntax. Please, let me remind you how JR
works with "current()":

  1. The expression "current()" looks like a function
    but it
    is not treated like one; it is not handled by the
    XPathFuncExpr class, so,
    if you try to use it alone you will get an error that
    the "function" is not
    supported. In other words, you cannot use it alone in
    an expression in order
    to refer to the "current" field.

  2. When an expression includes "current()", then it
    ALWAYS
    needs to be used in an XPath, actually, the
    expression needs to BE an XPath
    expression, so that it can be handled with the
    XPathPathExpr class. For that
    reason the expression NEEDS to contain at least one
    "/".

  3. Taking 1 and 2 into consideration, the ONLY way to
    refer to the "current" field (named "somefield") is
    to use "current()/." or
    "current()/somefield". So, I looks like that the
    original implementation was
    "okay" in the sense that the "current()" should only
    be used in XPaths.

If we wanted to have "current()" refer to the current
field, I believe we should introduce a new function
handler in XPathFuncExpr
class. Please let me know if this would work:

A. Whenever the "current()" expression is used inside
an
XPath, then the current code would be used as it is
used right now
(maintaining backwards compatibility), so that it
will still be referring to
the enclosing group, not the field itself.
B. Whenever the "current()" expression is used inside
a
non-XPath, then the new code would be used (in
XPathFuncExpr) and that would
be evaluated in a way that the current field would be
returned/referenced,
NOT the enclosing group.

That way, the following code could work, because it
wouldn't throw the "unrecognized function" error:

label <itemset

nodeset="instance('crop_list')/root/item[name =

current()]">
....
....

What do you think about that approach?

Meletis

On Friday, November 13, 2015 at 11:38:15 PM UTC+2, Martijn van de Rijdt wrote:

Hi,

I'd like to ask for a clarification about the
current()
node in an XPath predicate. It seems JR considers
this to be the parent node
of the current question. (Thanks to Chris for
sharing this.)

E.g. in this question below, current() equates to
/data/group and not to /data/group/options. I would
have expected it to be
the latter.

....
    <label>label</label>

    <itemset

nodeset="instance('crop_list')/root/item[name =
current()/../crop]">

         ....

         ....

    </itemset>

Is this by design? Or is it a bug?

We ran into this by realizing that the seemingly more
intuitive predicate [name = current()/../../crop] is
not working.

See this form that illustrates the behaviour nicely.
If
current() is used in a calculation (even if it makes
no sense to do so) this
extra step down is not taken, but in a predicate it
is. So it seems the
behaviour is not related to the current() function
but to how predicates are
handled in JR.

Thanks,
Martijn

--

--
Mitch Sundt
Software Engineer
University of Washington
mitche...@gmail.com

--

--
Mitch Sundt
Software Engineer
University of Washington
mitche...@gmail.com

--
Mitch Sundt
Software Engineer
University of Washington
mitche...@gmail.com

--
Mitch Sundt
Software Engineer
University of Washington
mitche...@gmail.com

--
You received this message because you are subscribed to the
Google Groups "ODK Developers" group.
To unsubscribe from this group and stop receiving emails
from
it, send an email to
opendatakit-developers+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--
Mitch Sundt
Software Engineer
University of Washington
mitche...@gmail.com

--
You received this message because you are subscribed to the
Google
Groups "ODK Developers" group.
To unsubscribe from this group and stop receiving emails from
it,
send an email to
opendatakit-developers+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the
Google
Groups "ODK Developers" group.
To unsubscribe from this group and stop receiving emails from
it,
send an email to
opendatakit-developers+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--

Mitch Sundt
Software Engineer
University of Washington
mitche...@gmail.com

--
Mitch Sundt
Software Engineer
University of Washington
mitche...@gmail.com

--
You received this message because you are subscribed to the Google
Groups "ODK Developers" group.
To unsubscribe from this group and stop receiving emails from it,
send
an email to opendatakit-developers+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--

You received this message because you are subscribed to a topic in
the
Google Groups "ODK Developers" group.
To unsubscribe from this topic, visit

https://groups.google.com/d/topic/opendatakit-developers/nN8HTA5pUIM/unsubscribe
.

To unsubscribe from this group and all its topics, send an email to
opendatakit-developers+unsubscribe@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--
Revolutionizing data collection since 2012.

Enketo | LinkedIn | GitHub | Twitter | Blog

--
You received this message because you are subscribed to the Google
Groups "ODK Developers" group.
To unsubscribe from this group and stop receiving emails from it,
send
an email to opendatakit-developers+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--

Mitch Sundt
Software Engineer
University of Washington

mitche...@gmail.com

--
You received this message because you are subscribed to the Google
Groups
"ODK Developers" group.
To unsubscribe from this group and stop receiving emails from it,
send an
email to opendatakit-developers+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google
Groups
"ODK Developers" group.
To unsubscribe from this group and stop receiving emails from it, send
an
email to opendatakit-developers+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google
Groups "ODK Developers" group.
To unsubscribe from this group and stop receiving emails from it, send
an email to opendatakit-developers+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to a topic in the
Google Groups "ODK Developers" group.
To unsubscribe from this topic, visit
https://groups.google.com/d/topic/opendatakit-developers/nN8HTA5pUIM/unsubscribe
.
To unsubscribe from this group and all its topics, send an email to
opendatakit-developers+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--
Revolutionizing data collection since 2012.

Enketo https://enketo.org/ | LinkedIn
http://www.linkedin.com/company/enketo-llc | GitHub
https://github.com/enketo | Twitter https://twitter.com/enketo
| Blog http://blog.enketo.org/

--
You received this message because you are subscribed to the Google Groups
"ODK Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to opendatakit-developers+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Ok, I was just wondering if it's really worth the effort. KoBo found 3
forms using current(). Not sure how many are actively used still. For them
it would seem to me easier to just contact that/those user(s) individually
to come up with a plan.

I can live with the XForm attribute and XLSForm setting addition.

Chris, to respond to your question about adding support for the bug to
Enketo. I don't currently see a reason to do so, as no Enketo user has ever
relied on the bug and I highly doubt any will in the future. However, I
don't think this should be relevant to this discussion. We're all in
agreement that we want the community to move forward with the correct
syntax.

··· On Tuesday, December 1, 2015 at 4:58:45 PM UTC-7, Christopher Robert wrote: > > We have individual users with 500+ devices in the field. Updating devices > -- or even keeping track of which devices are on which versions -- is a > major undertaking. In total, we're talking many thousands of devices spread > across many countries. The problem is not a small one, and our commitment > to backward compatibility is one that our users not only appreciate but > expect. > > ODK/SurveyCTO Collect continues to use older versions of forms to view, > edit, or finalize submissions begun under those older versions, even when > newer versions exist. So the backward-compatibility challenge is a > formidable one. If you change the way JavaRosa interprets expressions in > the kind of blanket way that you suggest, then users will be faced with the > impossible choice of running software that fails to function properly with > existing, in-progress, or old submissions, or running software that fails > to function properly with newer submissions. Multiply the difficulty of the > choice and the potential confusion of XPath parsing errors times > however-many devices there are in the field. > > No, this is a legitimately difficult situation that requires a tremendous > amount of care. Changing how XPath expressions are interpreted is a big > deal for the Collect world, though I can see why it would be considerably > less so in the Enketo world. > > Now, I will admit that there are bound to be very few users who already > use current() in choice filters. Mitch suggested the approach a few times > in the public forum, but even I failed to get it to work the first several > times I tried to follow his advice. So chances are that not many projects > are using current() today. (I know of 1-2, maybe there are 8-10 total. > That's just a guess.) > > Enketo aside, we could -- today! -- allow people to reference in-context > repeated values by exploiting the current JR support for current(). For > some users/projects, there would be an immediate benefit. But obviously > that relies on "the bug" and so it would expand the potential need for > backward compatibility. And it would break choice-filter compatibility > between Collect and Enketo (say, for SurveyCTO users at least). > > It's easy to say "just fix the bug and then this problem goes away," but > the fact is that we don't quite know how to fix it without breaking other > things; we have a candidate for a fix, but it's seriously scary. It may > take time. And meanwhile, we have already gone multiple years with users > waiting to be able to reference within-context repeated values in choice > filters. If a safe fix drags on for another year or two -- either because > it's just too hard or because nobody quite has the time -- is that just the > way it goes? I don't know. It's not obvious to me that breaking > compatibility with Enketo and relying on current behavior isn't the > welfare-maximizing choice. > > In the end, I believe that Mitch's suggestion re: compatibility is maybe > necessary (storing in the XML some indication of how current() in choice > filters should be interpreted). And if we totally screwed up the fix and > broke a bunch of use cases, at least it might offer some escape for > affected users (if we allowed an override on the settings sheet). > > So no, I don't agree that it's as simple as "fix the bug and make people > upgrade." > > Chris > > On Tue, Dec 1, 2015 at 5:11 PM Martijn van de Rijdt <mar...@enketo.org > wrote: > >> I can't help thinking that we're overthinking this >> backwards-compatibility issue. >> >> Ideally we have old clients that have the bug and new clients that do not >> have the bug. Isn't that the case with any other major bug fix? If an old >> form stops working in the new version of ODK Collect, the survey owner will >> either have to upload a new version of the form or the user will have to >> downgrade ODK Collect. Surely this will be a problem for some users, but >> for how many? A small fraction of one percent seems like a reasonable guess. >> >> Maybe the backwards-compatibility should be addressed just by warning >> users, as Yaw suggested way above. >> >> Any 'side-effects' of fixing the bug, like with creating itemsets, can be >> tested for, I presume, before deploying the fix. >> >> One way to detect at least a significant portion of the current current() >> users, is to do a regex search for any usage of the >> 'current()/path/to/node' where current()/ is not followed by '../'. That >> usage is specific to this bug to refer to a sibling node and will not occur >> any more after the bug is fixed (because a question cannot have children). >> >> On Mon, Nov 30, 2015 at 11:43 AM, Christopher Robert < cro...@surveycto.com > wrote: >> >>> I wasn't going to suggest adding the different parameters to any of the >>> XLSForm templates -- so they wouldn't become cluttered with this stuff. >>> Really, it would just be an emergency option for somebody requiring >>> different compatibility for some component that was out of sync with their >>> other components. Ideally, nobody would actually have to use the settings >>> sheet to override any of these options. >>> >>> To be specific, my proposal would be that everybody >>> support current_always_field=false today, and that this be how pyxform >>> today defaults. Ideally, this would include Enketo, as Collect and >>> SurveyCTO's Web Collect (and anything else using the ODK JavaRosa) already >>> support current_always_field=false as their current behavior. Then, we >>> figure out how to get current_always_field=true to work in JavaRosa and >>> phase that in, eventually changing the pyxform default to that once we >>> think that enough components support it. >>> >>> We have thousands of devices out there using older Collect versions, >>> even well after their servers have been updated... so my guess is that it's >>> some of those users who will end up having to use the settings-sheet option >>> to force current_always_field=false after we've changed the default to true. >>> >>> Does all of that make sense? >>> >>> Chris >>> >>> >>> On Sun, Nov 29, 2015 at 6:48 PM Yaw Anokwa <yan...@nafundi.com > wrote: >>> >>>> All this sounds fine to me, but wanted to raise some potential issues. >>>> Discrete parameters also means folks can pick and choose what behavior >>>> they want and it's hard to move the entire community forward that way. >>>> Also, once you get past two or three, you end up with a settings sheet >>>> of lots of flags to toggle and that isn't a great user experience. >>>> >>>> Yaw >>>> -- >>>> Need ODK consultants? Nafundi provides form design, server setup, >>>> in-field training, and software development for ODK. Go to >>>> https://nafundi.com to get started. >>>> >>>> On Sun, Nov 29, 2015 at 6:47 PM, Christopher Robert <cro...@surveycto.com > wrote: >>>> > Having a number of discrete parameters that are overridable on the >>>> settings >>>> > sheet makes sense to me (like "current_references_parent" or perhaps >>>> > "current_always_field"). We could then add model parameters and >>>> change the >>>> > pyxform default over time, but users could always override on their >>>> settings >>>> > sheets as needed for specific compatibility. >>>> > >>>> > For example, our SurveyCTO pyxform for 2.10 (we've given up on 2.01 >>>> now) >>>> > could default to current_always_field=false, which would mean that JR >>>> would >>>> > be backward-compatible and pyxform would emit relative paths for the >>>> > choice_filter column assuming that current() references the parent. >>>> This >>>> > would break all choice_filters in Enketo, but our Enketo users could >>>> set >>>> > current_always_field=true in their settings sheets. >>>> > >>>> > For that to work, though, we'd also have to add support for >>>> > current_always_field=true in JR. We could do that as Meletis has >>>> proposed, >>>> > which may have other side effects -- we'll have to see (e.g., for >>>> itemset >>>> > copying). >>>> > >>>> > Alternatively, we could try to convince Martijn to support >>>> > current_always_field=false, which is likely trivial. If he would do >>>> that, >>>> > then we could push JR's support for current_always_field=true to when >>>> we >>>> > could figure out how to do it safely. Also, it would mean producing a >>>> > version of Enketo that supports relative paths in choice filters in a >>>> manner >>>> > compatible with ODK Collect. Ona, Kobo, and other Enketo users could >>>> > rejoice, able to produce forms that use current() (like those that >>>> reference >>>> > same-repeat values) in their choice_filters. That boon to users has >>>> to be >>>> > worth something, even if a principle is slightly violated (supporting >>>> > something outside the XPath spec). >>>> > >>>> > Chris >>>> > >>>> > On Tue, Nov 24, 2015 at 8:43 PM Alex Dorey <dore...@gmail.com > wrote: >>>> >> >>>> >> Hi, Just catching up on the discussion. Sorry if I'm a bit behind in >>>> my >>>> >> understanding of things. >>>> >> >>>> >> Regarding the pyxform changes-- >>>> >> >>>> >> 1) Instead of the date, could we have the configuration parameter >>>> describe >>>> >> the specific change? >>>> >> >>>> >> So instead of: >>>> >> >>>> >> revision="2015-11-24" >>>> >> >>>> >> we specify something like: >>>> >> >>>> >> current_references_parent=True (default to False with the next >>>> pyxform >>>> >> version bump) >>>> >> >>>> >> >>>> >> 2) Could we put this configuration into the xlsform's settings >>>> sheet? I >>>> >> think this would be easier to test and configure. We could also add >>>> a way to >>>> >> override the setting with an extra argument to the CLI or xls2json >>>> call. >>>> >> >>>> >> >>>> >> >>>> >> On Tuesday, November 24, 2015 at 3:26:18 PM UTC-5, Mitch wrote: >>>> >>> >>>> >>> I agree. Having just delved into the itemset code, there are a >>>> number of >>>> >>> issues that would need to be cleaned up. >>>> >>> >>>> >>> What I would propose is to leave JR / ODK Collect broken as-is. >>>> This >>>> >>> will be future work. >>>> >>> >>>> >>> The python code will then be the focus of the tweaks. >>>> >>> >>>> >>> (1) change: >>>> >>> >>>> >>> json_survey = >>>> >>> xls2json.workbook_to_json(json.loads(request.POST['workbookJson']), >>>> >>> form_name=form_name, warnings=warningsList) >>>> >>> >>>> >>> to add an optional revision argument: >>>> >>> >>>> >>> json_survey = >>>> >>> xls2json.workbook_to_json(json.loads(request.POST['workbookJson']), >>>> >>> form_name=form_name, warnings=warningsList, revision='2015-11-24') >>>> >>> >>>> >>> >>>> >>> (2) change that implementation to: >>>> >>> >>>> >>> (2a) store the revision string somewhere in the returned json >>>> object. >>>> >>> >>>> >>> (2b) expand ${fieldname} with the new relative path expansion only >>>> if the >>>> >>> passed-in revision is >= '2015-11-24' ; otherwise, use the >>>> absolute path >>>> >>> expansion. >>>> >>> >>>> >>> (3) change: >>>> >>> >>>> >>> create_survey_element_from_dict() >>>> >>> >>>> >>> to emit the revision value as an attribute on the element. >>>> >>> >>>> >>> =============== >>>> >>> The JR itemset bug will be fixed at a later date. >>>> >>> >>>> >>> When that is fixed, we can introduce a new revision string for that >>>> >>> behavior change. >>>> >>> >>>> >>> i.e., once we have the revision string moving through the system, >>>> we can >>>> >>> begin to support non-backward-compatible changes to behaviors in >>>> XLSForm and >>>> >>> JR form processing. >>>> >>> >>>> >>> For the time being, the ODK XLSForm converter will not supply a >>>> revision >>>> >>> argument, and not make use of this pyxform enhancement. Once the JR >>>> fix is >>>> >>> done, we will then support that new revision and the current >>>> behavior, >>>> >>> defaulting to the new behavior, but supporting users in generating >>>> the old >>>> >>> behavior. >>>> >>> >>>> >>> Mitch >>>> >>> >>>> >>> On Tue, Nov 24, 2015 at 8:44 AM, Martijn van de Rijdt < mar...@enketo.org> wrote: >>>> >>>> >>>> >>>> I appreciate the creative solution. It does sound like it will >>>> become a >>>> >>>> bit of a headache to manage and explain to users though. Whether >>>> it's worth >>>> >>>> it depends on how many active forms are out there that rely on the >>>> bug. We >>>> >>>> should get Ona's and KoBo's input on this too, I think. >>>> >>>> >>>> >>>> From Enketo's side, relative paths were never supported in >>>> >>>> choice-filters (to my embarrassment and surprise - which led to my >>>> >>>> question). As a result, there is no significant issue with >>>> backwards >>>> >>>> compatibility in Enketo once it is fixed in JR. Not sure how >>>> meaningful this >>>> >>>> is to ODK Collect, but no Enketo user has ever reported this lack >>>> of >>>> >>>> support. I think to date it's been too hard to craft such >>>> choice-filters. >>>> >>>> Because of the way Enketo is integrated into KoBo, Ona, Formhub, >>>> I'd be >>>> >>>> surprised if there were a significant amount of those users using >>>> relative >>>> >>>> choice-filters. >>>> >>>> >>>> >>>> I do indeed feel strongly about following the XForms 2.0 spec >>>> here. I >>>> >>>> think relative path support is going to be very useful in the >>>> future, and I >>>> >>>> agree that we need to look at the bigger picture and do this >>>> correctly, so >>>> >>>> it won't catch up with us. >>>> >>>> >>>> >>>> On Tue, Nov 24, 2015 at 4:24 AM, Christopher Robert wrote: >>>> >>>>> >>>> >>>>> Okay, well, this is turning into a bigger and bigger project and >>>> we may >>>> >>>>> well miss the window for our 2.01 maintenance release (and/or >>>> make the >>>> >>>>> changes so extensive and scary that we cannot in good conscience >>>> include >>>> >>>>> them in a maintenance release). >>>> >>>>> >>>> >>>>> However, your idea for making JR sensitive to the revision value >>>> >>>>> emitted by pyxform opens a new possibility: we could release the >>>> pyxform >>>> >>>>> change with 2.01 so that users can begin to benefit from the >>>> relative paths >>>> >>>>> (relying on today's current() behavior), then assume that when JR >>>> is "fixed" >>>> >>>>> it will only be fixed for XML emitted by newer pyxforms that >>>> would have >>>> >>>>> updated their paths to account for the fix (using your >>>> revision-value idea). >>>> >>>>> The primary drawback to this approach is that we would basically >>>> break all >>>> >>>>> cascading-select filter expressions for SurveyCTO+Enketo users >>>> (since >>>> >>>>> Martijn clearly wants to implement the "fixed" version of >>>> current() rather >>>> >>>>> than the JavaRosa-compatible version of current()). >>>> >>>>> >>>> >>>>> Meletis also found another issue relating to the evaluation >>>> context in >>>> >>>>> jr:choice-name() ( >>>> https://github.com/opendatakit/opendatakit/issues/1180), >>>> >>>>> so we have that to contend with as well. If we can't safely >>>> resolve the >>>> >>>>> context for jr:choice-name(), then that would present another >>>> major drawback >>>> >>>>> and further lessen our enthusiasm for this entire adventure. >>>> >>>>> >>>> >>>>> Best, >>>> >>>>> >>>> >>>>> Chris >>>> >>>>> >>>> >>>>> On Mon, Nov 23, 2015 at 5:45 PM Mitch Sundt wrote: >>>> >>>>>> >>>> >>>>>> And, if it isn't clear, to preserve backward-compatible >>>> operations, >>>> >>>>>> XLSForm should not expand ${fieldname} with a relative path >>>> unless the >>>> >>>>>> revision value is the newer one; if the revision is not >>>> specified, it would >>>> >>>>>> emit absolute paths like it currently does. >>>> >>>>>> >>>> >>>>>> I also switched to use a date value for the revision value >>>> >>>>>> ("2015-11-23") which allows simple lexical comparisons. >>>> >>>>>> >>>> >>>>>> On Mon, Nov 23, 2015 at 1:03 PM, Mitch Sundt < mitche...@gmail.com> wrote: >>>> >>>>>>> >>>> >>>>>>> w.r.t. backward compatibility >>>> >>>>>>> >>>> >>>>>>> I too am concerned. >>>> >>>>>>> >>>> >>>>>>> A proposed solution: >>>> >>>>>>> >>>> >>>>>>> 1. Add a command line argument or other mechanism to request >>>> that >>>> >>>>>>> XLSForm inject a 'revision="2015-11-23"' attribute into the >>>> element. >>>> >>>>>>> e.g., >>>> >>>>>>> >>>> >>>>>>> This is used to alter the behavior of the XForms engine to >>>> >>>>>>> accommodate backward-compatible bug behaviors. >>>> >>>>>>> >>>> >>>>>>> We can't identify the bugs (e.g, 'legacy="itemset-parent"'), >>>> since >>>> >>>>>>> that would break older forms run on the new ODK Collect >>>> releases. >>>> >>>>>>> We have to advance the revision code as bugs are fixed. >>>> >>>>>>> >>>> >>>>>>> I think an integer value is fine for this. >>>> >>>>>>> >>>> >>>>>>> I don't anticipate making a lot of revisions, and I think the >>>> >>>>>>> revisions would be sequentially applied (you can't >>>> pick-and-choose what bugs >>>> >>>>>>> you include). >>>> >>>>>>> >>>> >>>>>>> In the future, this would allow us to re-do the date-time >>>> behavior. >>>> >>>>>>> >>>> >>>>>>> 2. Modify the XLSForm tools (e.g., the online form generator) to >>>> >>>>>>> enable the user to select what revision they want to emit (much >>>> like we did >>>> >>>>>>> last year with the JR form eval choice setting within ODK >>>> Collect). We can >>>> >>>>>>> default to emitting the latest revision if the user does not >>>> make a choice. >>>> >>>>>>> >>>> >>>>>>> 3. Augment JR to look for this revision attribute and >>>> store/restore >>>> >>>>>>> it in the FormDef binary object. >>>> >>>>>>> >>>> >>>>>>> Based upon this value, alter the behavior of the JR itemset >>>> >>>>>>> evaluation to fix the path bug. >>>> >>>>>>> >>>> >>>>>>> 4. Modify ODK Validate to scan for syntax that may be impacted >>>> by a >>>> >>>>>>> revision (to advise whether to change something). Not sure how >>>> complex that >>>> >>>>>>> might be. May not be always possible? >>>> >>>>>>> >>>> >>>>>>> ------------- >>>> >>>>>>> w.r.t. impacts of change >>>> >>>>>>> >>>> >>>>>>> Thank you Meletis for researching when this change occurred. >>>> >>>>>>> >>>> >>>>>>> I will have to ponder what effect the parent/node change would >>>> have >>>> >>>>>>> on itemset copying. Hadn't thought that there was any >>>> relationship between >>>> >>>>>>> the two, but perhaps there is. >>>> >>>>>>> >>>> >>>>>>> Yaw also reported that the Cascading Select itemsets on >>>> >>>>>>> opendatakit.appspot.com are not working with the 1.4.7 rev >>>> 1053 release (and >>>> >>>>>>> they work but give odd 'null' resolutions on 1057). This is in >>>> a related >>>> >>>>>>> area of this same code, so I will hopefully gain a better >>>> understanding of >>>> >>>>>>> dynamic itemsets as I uncover the causes of that problem. >>>> >>>>>>> >>>> >>>>>>> ------------- >>>> >>>>>>> Mitch >>>> >>>>>>> >>>> >>>>>>> On Mon, Nov 23, 2015 at 5:59 AM, Christopher Robert wrote: >>>> >>>>>>>> >>>> >>>>>>>> So just to be clear, the "fix" we're testing here reverts the >>>> entire >>>> >>>>>>>> context within that populateDynamicChoices() function, to be >>>> the field >>>> >>>>>>>> instead of the parent -- so the effects are potentially much >>>> wider than just >>>> >>>>>>>> what current() returns. Potentially anything and everything >>>> that depends on >>>> >>>>>>>> context can change now, since the context will shift from >>>> parent to field. >>>> >>>>>>>> >>>> >>>>>>>> This means that we revert a change from 5 years ago and alter >>>> the >>>> >>>>>>>> context from now on. Neither Meletis nor I feel terribly >>>> confident that we >>>> >>>>>>>> can entirely predict the consequences to backward >>>> compatibility. We're >>>> >>>>>>>> testing the things we can think to test, but honestly it's XML >>>> users in the >>>> >>>>>>>> broader ODK community who face the greatest risks here; >>>> SurveyCTO users by >>>> >>>>>>>> and large use XLSForm and thus the potential XPath syntax >>>> consequences are >>>> >>>>>>>> sharply limited. >>>> >>>>>>>> >>>> >>>>>>>> Are you guys very sure that you want to take this path? I can >>>> tell >>>> >>>>>>>> you now that if we make this change and then there is some >>>> outcry after the >>>> >>>>>>>> next ODK Collect release, SurveyCTO Collect will very likely >>>> stick with the >>>> >>>>>>>> new change (since we will have changed pyxform and we'll then >>>> have too many >>>> >>>>>>>> backward-compatibility concerns to then reverse the change). >>>> >>>>>>>> >>>> >>>>>>>> Thanks, >>>> >>>>>>>> >>>> >>>>>>>> Chris >>>> >>>>>>>> >>>> >>>>>>>> >>>> >>>>>>>> >>>> >>>>>>>> On Mon, Nov 23, 2015 at 5:23 AM wrote: >>>> >>>>>>>>> >>>> >>>>>>>>> Just for the history's sake, I found the time where the >>>> behavior >>>> >>>>>>>>> changed. It was at March 11, 2010: >>>> >>>>>>>>> >>>> >>>>>>>>> >>>> >>>>>>>>> >>>> https://bitbucket.org/m.sundt/javarosa/commits/b41875b2c23c9a21a868a5d2dc0ffc1f6b5b0b65#Lcore/src/org/javarosa/core/model/FormDef.javaT635 >>>> >>>>>>>>> >>>> >>>>>>>>> Take a look only in the changes of the >>>> "populateDynamicChoices" >>>> >>>>>>>>> method. The context reference used to be the "current" field, >>>> but after the >>>> >>>>>>>>> changes above it started be the field's parent (by adding >>>> getParent()). >>>> >>>>>>>>> Perhaps there was a reason for that change, but perhaps not, >>>> but I think all >>>> >>>>>>>>> of us agree that this should be fixed, even after 5 and a >>>> half years. >>>> >>>>>>>>> >>>> >>>>>>>>> Meletis >>>> >>>>>>>>> >>>> >>>>>>>>> On Monday, November 23, 2015 at 12:08:47 PM UTC+2, mel...@surveycto.com wrote: >>>> >>>>>>>>>> >>>> >>>>>>>>>> Hi Mitch, >>>> >>>>>>>>>> >>>> >>>>>>>>>> Please take a look at the code comment in line 31 here: >>>> >>>>>>>>>> >>>> >>>>>>>>>> >>>> >>>>>>>>>> >>>> https://bitbucket.org/m.sundt/javarosa/src/bf17fb749b74e53915f086ca3827c3c6fe73fd07/core/src/org/javarosa/core/model/ItemsetBinding.java?at=default&fileviewer=file-view-default#ItemsetBinding.java-31 >>>> >>>>>>>>>> >>>> >>>>>>>>>> It clearly says: "ref of the control parent (group/formdef) >>>> of >>>> >>>>>>>>>> itemset question" >>>> >>>>>>>>>> >>>> >>>>>>>>>> So, for some reason, they wanted it to point to the parent. >>>> I also >>>> >>>>>>>>>> see that when the dynamic choices are populated, the >>>> evaluation context is >>>> >>>>>>>>>> wrapped around that parent (contextRef object), that's why >>>> current() refers >>>> >>>>>>>>>> to the parent group when evaluating itemsets nodesets. >>>> >>>>>>>>>> >>>> >>>>>>>>>> I will run a few tests by changing the contextRef field >>>> value, I >>>> >>>>>>>>>> just wanted to mention that in case you remember a reason >>>> for that code >>>> >>>>>>>>>> comment. >>>> >>>>>>>>>> >>>> >>>>>>>>>> Meletis >>>> >>>>>>>>>> >>>> >>>>>>>>>> On Friday, November 20, 2015 at 8:49:12 PM UTC+2, Mitch wrote: >>>> >>>>>>>>>>> >>>> >>>>>>>>>>> Yes, I think it should be a one-line change; just don't know >>>> >>>>>>>>>>> where. >>>> >>>>>>>>>>> >>>> >>>>>>>>>>> Mitch >>>> >>>>>>>>>>> >>>> >>>>>>>>>>> (I do jar-updates over Christmas / New-Years. Tedious, but >>>> >>>>>>>>>>> doesn't require a whole lot of attention.) >>>> >>>>>>>>>>> >>>> >>>>>>>>>>> >>>> >>>>>>>>>>> >>>> >>>>>>>>>>> On Fri, Nov 20, 2015 at 10:36 AM, wrote: >>>> >>>>>>>>>>>> >>>> >>>>>>>>>>>> Hi Mitch, >>>> >>>>>>>>>>>> >>>> >>>>>>>>>>>> Thank you for all the details, I guess I was missing some >>>> of the >>>> >>>>>>>>>>>> context of the original issue. >>>> >>>>>>>>>>>> >>>> >>>>>>>>>>>> We will give it a shot to see if we can come up with a few >>>> ideas >>>> >>>>>>>>>>>> how to fix the evaluation content of the itemset nodeset. >>>> I just hope it >>>> >>>>>>>>>>>> will not be like last year with the "form processing >>>> logic" in JR, otherwise >>>> >>>>>>>>>>>> I believe that every Christmas will be a JR Christmas. :) >>>> >>>>>>>>>>>> >>>> >>>>>>>>>>>> Meletis >>>> >>>>>>>>>>>> >>>> >>>>>>>>>>>> >>>> >>>>>>>>>>>> On Friday, November 20, 2015 at 7:29:14 PM UTC+2, Mitch wrote: >>>> >>>>>>>>>>>>> >>>> >>>>>>>>>>>>> The form was attached in one of my earlier replies. >>>> >>>>>>>>>>>>> >>>> >>>>>>>>>>>>> On Fri, Nov 20, 2015 at 9:27 AM, Mitch Sundt wrote: >>>> >>>>>>>>>>>>>> >>>> >>>>>>>>>>>>>> And you see this in constraint expressions where you >>>> write: >>>> >>>>>>>>>>>>>> >>>> >>>>>>>>>>>>>> . > 6 >>>> >>>>>>>>>>>>>> >>>> >>>>>>>>>>>>>> I.e., current()/. should refer to the field, and not the >>>> group >>>> >>>>>>>>>>>>>> containing the field, as it currently does when >>>> evaluating expressions in >>>> >>>>>>>>>>>>>> the itemset nodeset. >>>> >>>>>>>>>>>>>> >>>> >>>>>>>>>>>>>> On Fri, Nov 20, 2015 at 9:26 AM, Mitch Sundt wrote: >>>> >>>>>>>>>>>>>>> >>>> >>>>>>>>>>>>>>> The issue is that there is a discrepancy in the meaning >>>> of >>>> >>>>>>>>>>>>>>> '.' across the different contexts of: calculate, >>>> relevant, , and >>>> >>>>>>>>>>>>>>> these itemset nodeset expressions. >>>> >>>>>>>>>>>>>>> >>>> >>>>>>>>>>>>>>> In other usages, you need to reference this other field >>>> with >>>> >>>>>>>>>>>>>>> current()/../filterfield ; but in the nodeset >>>> expression, you would >>>> >>>>>>>>>>>>>>> reference it with current()/./filterfield >>>> >>>>>>>>>>>>>>> >>>> >>>>>>>>>>>>>>> I.e., the notion of '.' is different when used within a >>>> >>>>>>>>>>>>>>> nodeset expression and when used within a constraint >>>> expression or an >>>> >>>>>>>>>>>>>>> field. >>>> >>>>>>>>>>>>>>> >>>> >>>>>>>>>>>>>>> See the attached NBiggestOfSet form. For the label, it >>>> has: >>>> >>>>>>>>>>>>>>> >>>> >>>>>>>>>>>>>>> >>> ref="/NBiggestOfSet/education/edu_level"> >>>> >>>>>>>>>>>>>>> What is >>> >>>>>>>>>>>>>>> value="current()/../referred_attendee_name"/>'s level >>>> of education? >>>> >>>>>>>>>>>>>>> ... >>>> >>>>>>>>>>>>>>> >>>> >>>>>>>>>>>>>>> For a constraint, it has: >>>> >>>>>>>>>>>>>>> >>> >>>>>>>>>>>>>>> >>>> calculate="/NBiggestOfSet/attendee/name[position(..)=int(current()/../referred_attendee)]" >>>> >>>>>>>>>>>>>>> >>>> nodeset="/NBiggestOfSet/education/referred_attendee_name" readonly="true()" >>>> >>>>>>>>>>>>>>> type="string"/> >>>> >>>>>>>>>>>>>>> >>>> >>>>>>>>>>>>>>> >>>> >>>>>>>>>>>>>>> These each use the current()/../ to reference the group >>>> ( >>>> >>>>>>>>>>>>>>> /NBiggesetOfSet/education ). >>>> >>>>>>>>>>>>>>> >>>> >>>>>>>>>>>>>>> But, within the nodeset definition (if this form used >>>> one), >>>> >>>>>>>>>>>>>>> you would need to use >>>> current()/./referred_attendee_name to reference a >>>> >>>>>>>>>>>>>>> field in that group. >>>> >>>>>>>>>>>>>>> >>>> >>>>>>>>>>>>>>> >>>> >>>>>>>>>>>>>>> >>>> >>>>>>>>>>>>>>> >>>> >>>>>>>>>>>>>>> On Fri, Nov 20, 2015 at 8:22 AM, wrote: >>>> >>>>>>>>>>>>>>>> >>>> >>>>>>>>>>>>>>>> Hi Mitch, >>>> >>>>>>>>>>>>>>>> >>>> >>>>>>>>>>>>>>>> I think I'm missing something but I think that nothing >>>> is >>>> >>>>>>>>>>>>>>>> broken and that the only issue is that the expression >>>> >>>>>>>>>>>>>>>> current()/../filterfield in your example above should >>>> be just changed to >>>> >>>>>>>>>>>>>>>> current()/filterfield. If you make that change, >>>> everything should work >>>> >>>>>>>>>>>>>>>> because it will be resolved to: >>>> >>>>>>>>>>>>>>>> >>>> >>>>>>>>>>>>>>>> /data/group1/filterfield >>>> >>>>>>>>>>>>>>>> >>>> >>>>>>>>>>>>>>>> Why should a "../" be present there? We have used >>>> >>>>>>>>>>>>>>>> current()/somefield successfully in one of our test >>>> forms and it works fine. >>>> >>>>>>>>>>>>>>>> And I think it would also work great in your example. >>>> Wouldn't it? Or am I >>>> >>>>>>>>>>>>>>>> totally out of context? >>>> >>>>>>>>>>>>>>>> >>>> >>>>>>>>>>>>>>>> Meletis >>>> >>>>>>>>>>>>>>>> >>>> >>>>>>>>>>>>>>>> >>>> >>>>>>>>>>>>>>>> On Friday, November 20, 2015 at 5:35:16 PM UTC+2, Mitch wrote: >>>> >>>>>>>>>>>>>>>>> >>>> >>>>>>>>>>>>>>>>> Hi Meletis, >>>> >>>>>>>>>>>>>>>>> >>>> >>>>>>>>>>>>>>>>> That's not the issue -- >>>> >>>>>>>>>>>>>>>>> >>>> >>>>>>>>>>>>>>>>> The issue is that if you use a filter condition in the >>>> >>&g >>> >>>

w.r.t. moving forward...

Even if this particular change impacts a very small number of forms, we
should figure out how to implement non-backward-compatible changes so that
they are safe moving forward. We have no visibility into who uses ODK
Collect and ODK Aggregate and no reliable way to notify them of changes.

This could be a huge deal when or if we ever revise the JR engine to
properly handle timezones, dates and times, and date and time arithmetic.

Or if we find a need to change XLSForm to emit XForms features or XPath
functions that currently don't work (e.g., to use [1] or position(), or
XForms events, for example).

Note that these non-backward-compatible changes extend to not just XLSForm
and ODK Collect, but also to servers. You can use the same revision tag to
indicate new or different processing to be taken on the server.

··· ============= *Managing this change requires:*

I believe that this requires:

(1) the XML form definition is the place to define the
feature-enabling-labels that the form requires of the device and server
processing steps.

(2) when the JR engine or any tool implements non-backward-compatible
changes, it only activates them when it detects the presence of those
feature-enabling-label in the XML form definition. Omitting that
feature-enabling-label is a directive to "stay in the past".

(3) ODK Collect and other tools processing the XML form definition or the
XML submission documents should warn the user (e.g., pop up a Toast) if
they are running a version that does not recognize the
feature-enabling-labels in the form's revision tag (i.e., they are given a
form that indicates an advanced feature that they are unaware of) -- this
notification informs the user that they should upgrade that tool to
correctly process the directive. Alternatively, the tool could reject the
form, Force Close or abort, which is the current behavior when ODK
Aggregate or ODK Collect gets an XForms definition that uses a new,
unrecognized, function, etc. ;-<

=============
The stable upgrade path is to:

(1) update your form distribution and data aggregation server(s),
(2) update your collection devices (or collection processing server --
e.g., Enketo), and
(3) only then update the form generation tool to emit forms with the
non-backward-compatible change.

=============
Specifying the Revision settings:

I believe the XLS file should not contain a setting to select or specify
the revision
.

I believe this should be specified at the time the XLSForm converter is
executed on the source XLS file.
I.e., the web UI for these tools should have a spinner or other UI element
that enables users to choose to "stay in the past"

This forces users to actively stay in the past.

The ODK site will probably achieve this with a drop-down that allows the
user to specify, e.g.,

"full current functionality"
"generate a form definition compatible with ODK Collect 1.4.7 and earlier"

To allow all groups to independently advance the functionality within their
tool families, the combination of features that each of these drop-down
choices corresponds to should be documented by those groups.

I.e., I see a future where a form generated on Ona for Enketo may not be
compatible with stock ODK Collect unless an appropriate compatibility
drop-down is selected.

This could allow, for example, Survey CTO to implement and recoup costs for
JR date-time fixes or other XForms extensions within their ecosystem before
releasing that fix back to the wider community.

I do not see any value in offering a user the choice of which features are
enabled or disabled. I suppose some sites could offer that, but it seems
overly complex. Users really just need a way to specify compatibility to
whatever older version of the tools they are using.

==============
XLSForm functionality

I am fine with the XLSForm generator taking a comma-separated list of
features to enable, and passing that list through via the revision tag to
the the XML form definition.

Upon reflection, I think this is the only way to manage multiple
independent parallel contributions to the various codebases. It allows
different groups to add and enable different functionalities at different
rates.

We just need to maintain a global list of all of the feature labels that
every group is using so that we don't choose names that collide.

And, some of these feature labels may simply pass through the XLSForm
generator and on to the downstream tools. e.g., date-time fixes might not
have any impact on XLSForm, but would alter how the JR engine processed the
form.

On Wed, Dec 2, 2015 at 10:25 AM, Martijn van de Rijdt martijn@enketo.org wrote:

Ok, I was just wondering if it's really worth the effort. KoBo found 3
forms using current(). Not sure how many are actively used still. For them
it would seem to me easier to just contact that/those user(s) individually
to come up with a plan.

I can live with the XForm attribute and XLSForm setting addition.

Chris, to respond to your question about adding support for the bug to
Enketo. I don't currently see a reason to do so, as no Enketo user has ever
relied on the bug and I highly doubt any will in the future. However, I
don't think this should be relevant to this discussion. We're all in
agreement that we want the community to move forward with the correct
syntax.

On Tuesday, December 1, 2015 at 4:58:45 PM UTC-7, Christopher Robert wrote:

We have individual users with 500+ devices in the field. Updating devices
-- or even keeping track of which devices are on which versions -- is a
major undertaking. In total, we're talking many thousands of devices spread
across many countries. The problem is not a small one, and our commitment
to backward compatibility is one that our users not only appreciate but
expect.

ODK/SurveyCTO Collect continues to use older versions of forms to view,
edit, or finalize submissions begun under those older versions, even when
newer versions exist. So the backward-compatibility challenge is a
formidable one. If you change the way JavaRosa interprets expressions in
the kind of blanket way that you suggest, then users will be faced with the
impossible choice of running software that fails to function properly with
existing, in-progress, or old submissions, or running software that fails
to function properly with newer submissions. Multiply the difficulty of the
choice and the potential confusion of XPath parsing errors times
however-many devices there are in the field.

No, this is a legitimately difficult situation that requires a tremendous
amount of care. Changing how XPath expressions are interpreted is a big
deal for the Collect world, though I can see why it would be considerably
less so in the Enketo world.

Now, I will admit that there are bound to be very few users who already
use current() in choice filters. Mitch suggested the approach a few times
in the public forum, but even I failed to get it to work the first several
times I tried to follow his advice. So chances are that not many projects
are using current() today. (I know of 1-2, maybe there are 8-10 total.
That's just a guess.)

Enketo aside, we could -- today! -- allow people to reference in-context
repeated values by exploiting the current JR support for current(). For
some users/projects, there would be an immediate benefit. But obviously
that relies on "the bug" and so it would expand the potential need for
backward compatibility. And it would break choice-filter compatibility
between Collect and Enketo (say, for SurveyCTO users at least).

It's easy to say "just fix the bug and then this problem goes away," but
the fact is that we don't quite know how to fix it without breaking other
things; we have a candidate for a fix, but it's seriously scary. It may
take time. And meanwhile, we have already gone multiple years with users
waiting to be able to reference within-context repeated values in choice
filters. If a safe fix drags on for another year or two -- either because
it's just too hard or because nobody quite has the time -- is that just the
way it goes? I don't know. It's not obvious to me that breaking
compatibility with Enketo and relying on current behavior isn't the
welfare-maximizing choice.

In the end, I believe that Mitch's suggestion re: compatibility is maybe
necessary (storing in the XML some indication of how current() in choice
filters should be interpreted). And if we totally screwed up the fix and
broke a bunch of use cases, at least it might offer some escape for
affected users (if we allowed an override on the settings sheet).

So no, I don't agree that it's as simple as "fix the bug and make people
upgrade."

Chris

On Tue, Dec 1, 2015 at 5:11 PM Martijn van de Rijdt mar...@enketo.org wrote:

I can't help thinking that we're overthinking this
backwards-compatibility issue.

Ideally we have old clients that have the bug and new clients that do
not have the bug. Isn't that the case with any other major bug fix? If an
old form stops working in the new version of ODK Collect, the survey owner
will either have to upload a new version of the form or the user will have
to downgrade ODK Collect. Surely this will be a problem for some users, but
for how many? A small fraction of one percent seems like a reasonable guess.

Maybe the backwards-compatibility should be addressed just by warning
users, as Yaw suggested way above.

Any 'side-effects' of fixing the bug, like with creating itemsets, can
be tested for, I presume, before deploying the fix.

One way to detect at least a significant portion of the current
current() users, is to do a regex search for any usage of the
'current()/path/to/node' where current()/ is not followed by '../'. That
usage is specific to this bug to refer to a sibling node and will not occur
any more after the bug is fixed (because a question cannot have children).

On Mon, Nov 30, 2015 at 11:43 AM, Christopher Robert < cro...@surveycto.com> wrote:

I wasn't going to suggest adding the different parameters to any of the
XLSForm templates -- so they wouldn't become cluttered with this stuff.
Really, it would just be an emergency option for somebody requiring
different compatibility for some component that was out of sync with their
other components. Ideally, nobody would actually have to use the settings
sheet to override any of these options.

To be specific, my proposal would be that everybody
support current_always_field=false today, and that this be how pyxform
today defaults. Ideally, this would include Enketo, as Collect and
SurveyCTO's Web Collect (and anything else using the ODK JavaRosa) already
support current_always_field=false as their current behavior. Then, we
figure out how to get current_always_field=true to work in JavaRosa and
phase that in, eventually changing the pyxform default to that once we
think that enough components support it.

We have thousands of devices out there using older Collect versions,
even well after their servers have been updated... so my guess is that it's
some of those users who will end up having to use the settings-sheet option
to force current_always_field=false after we've changed the default to true.

Does all of that make sense?

Chris

On Sun, Nov 29, 2015 at 6:48 PM Yaw Anokwa yan...@nafundi.com wrote:

All this sounds fine to me, but wanted to raise some potential issues.
Discrete parameters also means folks can pick and choose what behavior
they want and it's hard to move the entire community forward that way.
Also, once you get past two or three, you end up with a settings sheet
of lots of flags to toggle and that isn't a great user experience.

Yaw

Need ODK consultants? Nafundi provides form design, server setup,
in-field training, and software development for ODK. Go to
https://nafundi.com to get started.

On Sun, Nov 29, 2015 at 6:47 PM, Christopher Robert cro...@surveycto.com wrote:

Having a number of discrete parameters that are overridable on the
settings
sheet makes sense to me (like "current_references_parent" or perhaps
"current_always_field"). We could then add model parameters and
change the
pyxform default over time, but users could always override on their
settings
sheets as needed for specific compatibility.

For example, our SurveyCTO pyxform for 2.10 (we've given up on 2.01
now)
could default to current_always_field=false, which would mean that
JR would
be backward-compatible and pyxform would emit relative paths for the
choice_filter column assuming that current() references the parent.
This
would break all choice_filters in Enketo, but our Enketo users could
set
current_always_field=true in their settings sheets.

For that to work, though, we'd also have to add support for
current_always_field=true in JR. We could do that as Meletis has
proposed,
which may have other side effects -- we'll have to see (e.g., for
itemset
copying).

Alternatively, we could try to convince Martijn to support
current_always_field=false, which is likely trivial. If he would do
that,
then we could push JR's support for current_always_field=true to
when we
could figure out how to do it safely. Also, it would mean producing a
version of Enketo that supports relative paths in choice filters in
a manner
compatible with ODK Collect. Ona, Kobo, and other Enketo users could
rejoice, able to produce forms that use current() (like those that
reference
same-repeat values) in their choice_filters. That boon to users has
to be
worth something, even if a principle is slightly violated (supporting
something outside the XPath spec).

Chris

On Tue, Nov 24, 2015 at 8:43 PM Alex Dorey dore...@gmail.com wrote:

Hi, Just catching up on the discussion. Sorry if I'm a bit behind
in my
understanding of things.

Regarding the pyxform changes--

  1. Instead of the date, could we have the configuration parameter
    describe
    the specific change?

So instead of:

revision="2015-11-24"

we specify something like:

current_references_parent=True (default to False with the next
pyxform
version bump)

  1. Could we put this configuration into the xlsform's settings
    sheet? I
    think this would be easier to test and configure. We could also add
    a way to
    override the setting with an extra argument to the CLI or xls2json
    call.

On Tuesday, November 24, 2015 at 3:26:18 PM UTC-5, Mitch wrote:

I agree. Having just delved into the itemset code, there are a
number of
issues that would need to be cleaned up.

What I would propose is to leave JR / ODK Collect broken as-is.
This
will be future work.

The python code will then be the focus of the tweaks.

(1) change:

json_survey =
xls2json.workbook_to_json(json.loads(request.POST['workbookJson']),
form_name=form_name, warnings=warningsList)

to add an optional revision argument:

json_survey =
xls2json.workbook_to_json(json.loads(request.POST['workbookJson']),
form_name=form_name, warnings=warningsList, revision='2015-11-24')

(2) change that implementation to:

(2a) store the revision string somewhere in the returned json
object.

(2b) expand ${fieldname} with the new relative path expansion only
if the
passed-in revision is >= '2015-11-24' ; otherwise, use the
absolute path
expansion.

(3) change:

create_survey_element_from_dict()

to emit the revision value as an attribute on the element.

===============
The JR itemset bug will be fixed at a later date.

When that is fixed, we can introduce a new revision string for that
behavior change.

i.e., once we have the revision string moving through the system,
we can
begin to support non-backward-compatible changes to behaviors in
XLSForm and
JR form processing.

For the time being, the ODK XLSForm converter will not supply a
revision
argument, and not make use of this pyxform enhancement. Once the
JR fix is
done, we will then support that new revision and the current
behavior,
defaulting to the new behavior, but supporting users in generating
the old
behavior.

Mitch

On Tue, Nov 24, 2015 at 8:44 AM, Martijn van de Rijdt < mar...@enketo.org> wrote:

I appreciate the creative solution. It does sound like it will
become a
bit of a headache to manage and explain to users though. Whether
it's worth
it depends on how many active forms are out there that rely on
the bug. We
should get Ona's and KoBo's input on this too, I think.

From Enketo's side, relative paths were never supported in
choice-filters (to my embarrassment and surprise - which led to my
question). As a result, there is no significant issue with
backwards
compatibility in Enketo once it is fixed in JR. Not sure how
meaningful this
is to ODK Collect, but no Enketo user has ever reported this lack
of
support. I think to date it's been too hard to craft such
choice-filters.
Because of the way Enketo is integrated into KoBo, Ona, Formhub,
I'd be
surprised if there were a significant amount of those users using
relative
choice-filters.

I do indeed feel strongly about following the XForms 2.0 spec
here. I
think relative path support is going to be very useful in the
future, and I
agree that we need to look at the bigger picture and do this
correctly, so
it won't catch up with us.

On Tue, Nov 24, 2015 at 4:24 AM, Christopher Robert cro...@surveycto.com wrote:

Okay, well, this is turning into a bigger and bigger project and
we may
well miss the window for our 2.01 maintenance release (and/or
make the
changes so extensive and scary that we cannot in good conscience
include
them in a maintenance release).

However, your idea for making JR sensitive to the revision value
emitted by pyxform opens a new possibility: we could release the
pyxform
change with 2.01 so that users can begin to benefit from the
relative paths
(relying on today's current() behavior), then assume that when
JR is "fixed"
it will only be fixed for XML emitted by newer pyxforms that
would have
updated their paths to account for the fix (using your
revision-value idea).
The primary drawback to this approach is that we would basically
break all
cascading-select filter expressions for SurveyCTO+Enketo users
(since
Martijn clearly wants to implement the "fixed" version of
current() rather
than the JavaRosa-compatible version of current()).

Meletis also found another issue relating to the evaluation
context in
jr:choice-name() (
https://github.com/opendatakit/opendatakit/issues/1180),
so we have that to contend with as well. If we can't safely
resolve the
context for jr:choice-name(), then that would present another
major drawback
and further lessen our enthusiasm for this entire adventure.

Best,

Chris

On Mon, Nov 23, 2015 at 5:45 PM Mitch Sundt <mitche...@gmail.com wrote:

And, if it isn't clear, to preserve backward-compatible
operations,
XLSForm should not expand ${fieldname} with a relative path
unless the
revision value is the newer one; if the revision is not
specified, it would
emit absolute paths like it currently does.

I also switched to use a date value for the revision value
("2015-11-23") which allows simple lexical comparisons.

On Mon, Nov 23, 2015 at 1:03 PM, Mitch Sundt < mitche...@gmail.com> wrote:

w.r.t. backward compatibility

I too am concerned.

A proposed solution:

  1. Add a command line argument or other mechanism to request
    that
    XLSForm inject a 'revision="2015-11-23"' attribute into the
    element.
    e.g.,

This is used to alter the behavior of the XForms engine to
accommodate backward-compatible bug behaviors.

We can't identify the bugs (e.g, 'legacy="itemset-parent"'),
since
that would break older forms run on the new ODK Collect
releases.
We have to advance the revision code as bugs are fixed.

I think an integer value is fine for this.

I don't anticipate making a lot of revisions, and I think the
revisions would be sequentially applied (you can't
pick-and-choose what bugs
you include).

In the future, this would allow us to re-do the date-time
behavior.

  1. Modify the XLSForm tools (e.g., the online form generator)
    to
    enable the user to select what revision they want to emit
    (much like we did
    last year with the JR form eval choice setting within ODK
    Collect). We can
    default to emitting the latest revision if the user does not
    make a choice.

  2. Augment JR to look for this revision attribute and
    store/restore
    it in the FormDef binary object.

Based upon this value, alter the behavior of the JR itemset
evaluation to fix the path bug.

  1. Modify ODK Validate to scan for syntax that may be impacted
    by a
    revision (to advise whether to change something). Not sure
    how complex that
    might be. May not be always possible?

w.r.t. impacts of change

Thank you Meletis for researching when this change occurred.

I will have to ponder what effect the parent/node change would
have
on itemset copying. Hadn't thought that there was any
relationship between
the two, but perhaps there is.

Yaw also reported that the Cascading Select itemsets on
opendatakit.appspot.com are not working with the 1.4.7 rev
1053 release (and
they work but give odd 'null' resolutions on 1057). This is in
a related
area of this same code, so I will hopefully gain a better
understanding of
dynamic itemsets as I uncover the causes of that problem.


Mitch

On Mon, Nov 23, 2015 at 5:59 AM, Christopher Robert cro...@surveycto.com wrote:

So just to be clear, the "fix" we're testing here reverts the
entire
context within that populateDynamicChoices() function, to be
the field
instead of the parent -- so the effects are potentially much
wider than just
what current() returns. Potentially anything and everything
that depends on
context can change now, since the context will shift from
parent to field.

This means that we revert a change from 5 years ago and alter
the
context from now on. Neither Meletis nor I feel terribly
confident that we
can entirely predict the consequences to backward
compatibility. We're
testing the things we can think to test, but honestly it's
XML users in the
broader ODK community who face the greatest risks here;
SurveyCTO users by
and large use XLSForm and thus the potential XPath syntax
consequences are
sharply limited.

Are you guys very sure that you want to take this path? I can
tell
you now that if we make this change and then there is some
outcry after the
next ODK Collect release, SurveyCTO Collect will very likely
stick with the
new change (since we will have changed pyxform and we'll then
have too many
backward-compatibility concerns to then reverse the change).

Thanks,

Chris

On Mon, Nov 23, 2015 at 5:23 AM mel...@surveycto.com wrote:

Just for the history's sake, I found the time where the
behavior
changed. It was at March 11, 2010:

https://bitbucket.org/m.sundt/javarosa/commits/b41875b2c23c9a21a868a5d2dc0ffc1f6b5b0b65#Lcore/src/org/javarosa/core/model/FormDef.javaT635

Take a look only in the changes of the
"populateDynamicChoices"
method. The context reference used to be the "current"
field, but after the
changes above it started be the field's parent (by adding
getParent()).
Perhaps there was a reason for that change, but perhaps not,
but I think all
of us agree that this should be fixed, even after 5 and a
half years.

Meletis

On Monday, November 23, 2015 at 12:08:47 PM UTC+2, mel...@surveycto.com wrote:

Hi Mitch,

Please take a look at the code comment in line 31 here:

https://bitbucket.org/m.sundt/javarosa/src/bf17fb749b74e53915f086ca3827c3c6fe73fd07/core/src/org/javarosa/core/model/ItemsetBinding.java?at=default&fileviewer=file-view-default#ItemsetBinding.java-31

It clearly says: "ref of the control parent (group/formdef)
of
itemset question"

So, for some reason, they wanted it to point to the parent.
I also
see that when the dynamic choices are populated, the
evaluation context is
wrapped around that parent (contextRef object), that's why
current() refers
to the parent group when evaluating itemsets nodesets.

I will run a few tests by changing the contextRef field
value, I
just wanted to mention that in case you remember a reason
for that code
comment.

Meletis

On Friday, November 20, 2015 at 8:49:12 PM UTC+2, Mitch wrote:

Yes, I think it should be a one-line change; just don't
know
where.

Mitch

(I do jar-updates over Christmas / New-Years. Tedious, but
doesn't require a whole lot of attention.)

On Fri, Nov 20, 2015 at 10:36 AM, mel...@surveycto.com wrote:

Hi Mitch,

Thank you for all the details, I guess I was missing some
of the
context of the original issue.

We will give it a shot to see if we can come up with a
few ideas
how to fix the evaluation content of the itemset nodeset.
I just hope it
will not be like last year with the "form processing
logic" in JR, otherwise
I believe that every Christmas will be a JR Christmas. :slight_smile:

Meletis

On Friday, November 20, 2015 at 7:29:14 PM UTC+2, Mitch wrote:

The form was attached in one of my earlier replies.

On Fri, Nov 20, 2015 at 9:27 AM, Mitch Sundt mitche...@gmail.com wrote:

And you see this in constraint expressions where you
write:

. > 6

I.e., current()/. should refer to the field, and not
the group
containing the field, as it currently does when
evaluating expressions in
the itemset nodeset.

On Fri, Nov 20, 2015 at 9:26 AM, Mitch Sundt mitche...@gmail.com wrote:

The issue is that there is a discrepancy in the
meaning of
'.' across the different contexts of: calculate,
relevant, , and
these itemset nodeset expressions.

In other usages, you need to reference this other
field with
current()/../filterfield ; but in the nodeset
expression, you would
reference it with current()/./filterfield

I.e., the notion of '.' is different when used within a
nodeset expression and when used within a constraint
expression or an
field.

See the attached NBiggestOfSet form. For the label,
it has:

    <select1

ref="/NBiggestOfSet/education/edu_level">

      <label>What is <output

value="current()/../referred_attendee_name"/>'s level
of education?
...

For a constraint, it has:
<bind

calculate="/NBiggestOfSet/attendee/name[position(..)=int(current()/../referred_attendee)]"

nodeset="/NBiggestOfSet/education/referred_attendee_name" readonly="true()"

type="string"/>

These each use the current()/../ to reference the
group (
/NBiggesetOfSet/education ).

But, within the nodeset definition (if this form used
one),
you would need to use
current()/./referred_attendee_name to reference a
field in that group.

On Fri, Nov 20, 2015 at 8:22 AM, <mel...@surveycto.com wrote:

Hi Mitch,

I think I'm missing something but I think that
nothing is
broken and that the only issue is that the expression
current()/../filterfield in your example above should
be just changed to
current()/filterfield. If you make that change,
everything should work
because it will be resolved to:

/data/group1/filterfield

Why should a "../" be present there? We have used
current()/somefield successfully in one of our test
forms and it works fine.
And I think it would also work great in your example.
Wouldn't it? Or am I
totally out of context?

Meletis

On Friday, November 20, 2015 at 5:35:16 PM UTC+2, Mitch wrote:

Hi Meletis,

That's not the issue --

The issue is that if you use a filter condition in
the
&g

--
Revolutionizing data collection since 2012.

Enketo https://enketo.org/ | LinkedIn
http://www.linkedin.com/company/enketo-llc | GitHub
https://github.com/enketo | Twitter https://twitter.com/enketo
| Blog http://blog.enketo.org/

--
You received this message because you are subscribed to the Google Groups
"ODK Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to opendatakit-developers+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--
Mitch Sundt
Software Engineer
University of Washington
mitchellsundt@gmail.com

Thanks all, I appreciate all of the thinking going into this.

At the risk of thoroughly exhausting everybody's patience, I'm still not
sure that we have a strategy that can be described as reasonable from an
existing user's point of view. Let me explain why.

Say we release SurveyCTO 2.10 with three changes:

  1. We add the fix to JR and have it apply whenever the appropriate flag is
    set in the XForm model.

  2. We default all pyxform generation (which is basically invisible to the
    user in SurveyCTO) to include the appropriate flag so that newly-generated
    XML will be interpreted with the fixed JR.

  3. We default pyxform to output XPaths in choice-filter expressions using
    current() so that within-context repeat references finally work. This code,
    by default, relies on the fix (so paths are relative to the field, not the
    group). (It also changes ALL XPath expressions, not just those referencing
    within-repeat values, because we're not clever enough to be able to
    distinguish the different cases in that python code.)

Now say you're an upgrading user and you upgraded because SurveyCTO 2.10
also included a beautiful and astonishingly powerful web-based form
designer. You upgraded to get this new designer and you love it and now you
upload a new or revised form. That form happens to have a choice-filter
expression. Now here's where things get ugly...

That form won't work on your SurveyCTO 2.01, 2.00, 1.402, etc. devices.
Probably there's some nasty XPath parsing error when you get to where the
choice filter kicks in, maybe it even crashes your Collect. That form might
have worked the day before, and maybe all you changed was fixing a typo in
a label, but now suddenly it doesn't work on all of these devices.

So here's where we say, okay, the user has two choices: (1) upgrade all
devices to the 2.10 Collect, or (2) find and set the "stay in the past"
option. The second option is good in principle -- because it provides a
kind of quick save -- but then what does it take to ever migrate away from
that "stay in the past" setting? Not only do you need to then update 100%
of your devices with a high degree of certainty, but you also need to
re-generate the XML for every form on your server. And not even that is
enough. Then you need to update every form definition on every device.

And if you miss a few devices and leave them on the old version, or miss a
few form definitions here or there? Cryptic errors, inability to finalize
forms, confusion, anger, frantic calls to support, etc., etc.

You don't need to be a human-centered design expert to see that this is
kind of a disaster from a user-experience point of view. And, for us, this
would be multiplied across hundreds of migrating users, essentially eating
a lot of our 2016 and causing a tremendous amount of ill will.

The only upgrade path for existing users that makes any sense is one where
we basically emit the "stay in the past" flag always for everyone. You can
all turn your noses up at the idea of indefinite reliance on what was
initially a bug, and I get it. But at least here at SurveyCTO, we're not
going to put our users through the kind of migration path described above.

If relying on the bug is just too unpalatable, then ultimately I don't
think that we can change pyxform to emit XPaths using current(). It's just
too fraught and we don't have a good way to manage the upgrade path without
torturing existing users. So fine, our awesome pyxform fix to magically
make within-repeat references start working is just not destined to see the
light of day. Rather, we need to make pyxform smart enough to only use
current() for paths to repeated values (which never worked in old JR's
anyway). Or we need to just provide a new solution for referencing repeated
values without current() (perhaps a new custom JR function, which everybody
loves).

And in that case, sure, I'm much more relaxed about the strategies for
upgrading users since very few are likely to be using current(). I won't
want to overly torture those users, but I won't lose a ton of sleep over
them since even users who use current() now are unlikely to use it in a lot
of forms.

I'll see if we can figure out how to make the pyxform change in such a way
that it only changes the XPaths for fields within repeat groups. But I
might also see if we can instead find a totally different way to solve or
work around this problem. Or, I'll see how I feel about pushing this off
for another year. Fixing choice filters to be able to work will within
repeat groups was not meant to be a huge project.

Thanks again,

Chris

··· On Wed, Dec 2, 2015 at 2:48 PM Mitch Sundt wrote:

w.r.t. moving forward...

Even if this particular change impacts a very small number of forms, we
should figure out how to implement non-backward-compatible changes so that
they are safe moving forward. We have no visibility into who uses ODK
Collect and ODK Aggregate and no reliable way to notify them of changes.

This could be a huge deal when or if we ever revise the JR engine to
properly handle timezones, dates and times, and date and time arithmetic.

Or if we find a need to change XLSForm to emit XForms features or XPath
functions that currently don't work (e.g., to use [1] or position(), or
XForms events, for example).

Note that these non-backward-compatible changes extend to not just XLSForm
and ODK Collect, but also to servers. You can use the same revision tag to
indicate new or different processing to be taken on the server.

=============
Managing this change requires:

I believe that this requires:

(1) the XML form definition is the place to define the
feature-enabling-labels that the form requires of the device and server
processing steps.

(2) when the JR engine or any tool implements non-backward-compatible
changes, it only activates them when it detects the presence of those
feature-enabling-label in the XML form definition. Omitting that
feature-enabling-label is a directive to "stay in the past".

(3) ODK Collect and other tools processing the XML form definition or the
XML submission documents should warn the user (e.g., pop up a Toast) if
they are running a version that does not recognize the
feature-enabling-labels in the form's revision tag (i.e., they are given a
form that indicates an advanced feature that they are unaware of) -- this
notification informs the user that they should upgrade that tool to
correctly process the directive. Alternatively, the tool could reject the
form, Force Close or abort, which is the current behavior when ODK
Aggregate or ODK Collect gets an XForms definition that uses a new,
unrecognized, function, etc. ;-<

=============
The stable upgrade path is to:

(1) update your form distribution and data aggregation server(s),
(2) update your collection devices (or collection processing server --
e.g., Enketo), and
(3) only then update the form generation tool to emit forms with the
non-backward-compatible change.

=============
Specifying the Revision settings:

I believe the XLS file should not contain a setting to select or specify
the revision
.

I believe this should be specified at the time the XLSForm converter is
executed on the source XLS file.
I.e., the web UI for these tools should have a spinner or other UI element
that enables users to choose to "stay in the past"

This forces users to actively stay in the past.

The ODK site will probably achieve this with a drop-down that allows the
user to specify, e.g.,

"full current functionality"
"generate a form definition compatible with ODK Collect 1.4.7 and earlier"

To allow all groups to independently advance the functionality within
their tool families, the combination of features that each of these
drop-down choices corresponds to should be documented by those groups.

I.e., I see a future where a form generated on Ona for Enketo may not be
compatible with stock ODK Collect unless an appropriate compatibility
drop-down is selected.

This could allow, for example, Survey CTO to implement and recoup costs
for JR date-time fixes or other XForms extensions within their ecosystem
before releasing that fix back to the wider community.

I do not see any value in offering a user the choice of which features are
enabled or disabled. I suppose some sites could offer that, but it seems
overly complex. Users really just need a way to specify compatibility to
whatever older version of the tools they are using.

==============
XLSForm functionality

I am fine with the XLSForm generator taking a comma-separated list of
features to enable, and passing that list through via the revision tag to
the the XML form definition.

Upon reflection, I think this is the only way to manage multiple
independent parallel contributions to the various codebases. It allows
different groups to add and enable different functionalities at different
rates.

We just need to maintain a global list of all of the feature labels that
every group is using so that we don't choose names that collide.

And, some of these feature labels may simply pass through the XLSForm
generator and on to the downstream tools. e.g., date-time fixes might not
have any impact on XLSForm, but would alter how the JR engine processed the
form.

On Wed, Dec 2, 2015 at 10:25 AM, Martijn van de Rijdt martijn@enketo.org wrote:

Ok, I was just wondering if it's really worth the effort. KoBo found 3
forms using current(). Not sure how many are actively used still. For them
it would seem to me easier to just contact that/those user(s) individually
to come up with a plan.

I can live with the XForm attribute and XLSForm setting addition.

Chris, to respond to your question about adding support for the bug to
Enketo. I don't currently see a reason to do so, as no Enketo user has ever
relied on the bug and I highly doubt any will in the future. However, I
don't think this should be relevant to this discussion. We're all in
agreement that we want the community to move forward with the correct
syntax.

On Tuesday, December 1, 2015 at 4:58:45 PM UTC-7, Christopher Robert wrote:

We have individual users with 500+ devices in the field. Updating
devices -- or even keeping track of which devices are on which versions --
is a major undertaking. In total, we're talking many thousands of devices
spread across many countries. The problem is not a small one, and our
commitment to backward compatibility is one that our users not only
appreciate but expect.

ODK/SurveyCTO Collect continues to use older versions of forms to view,
edit, or finalize submissions begun under those older versions, even when
newer versions exist. So the backward-compatibility challenge is a
formidable one. If you change the way JavaRosa interprets expressions in
the kind of blanket way that you suggest, then users will be faced with the
impossible choice of running software that fails to function properly with
existing, in-progress, or old submissions, or running software that fails
to function properly with newer submissions. Multiply the difficulty of the
choice and the potential confusion of XPath parsing errors times
however-many devices there are in the field.

No, this is a legitimately difficult situation that requires a
tremendous amount of care. Changing how XPath expressions are interpreted
is a big deal for the Collect world, though I can see why it would be
considerably less so in the Enketo world.

Now, I will admit that there are bound to be very few users who already
use current() in choice filters. Mitch suggested the approach a few times
in the public forum, but even I failed to get it to work the first several
times I tried to follow his advice. So chances are that not many projects
are using current() today. (I know of 1-2, maybe there are 8-10 total.
That's just a guess.)

Enketo aside, we could -- today! -- allow people to reference in-context
repeated values by exploiting the current JR support for current(). For
some users/projects, there would be an immediate benefit. But obviously
that relies on "the bug" and so it would expand the potential need for
backward compatibility. And it would break choice-filter compatibility
between Collect and Enketo (say, for SurveyCTO users at least).

It's easy to say "just fix the bug and then this problem goes away," but
the fact is that we don't quite know how to fix it without breaking other
things; we have a candidate for a fix, but it's seriously scary. It may
take time. And meanwhile, we have already gone multiple years with users
waiting to be able to reference within-context repeated values in choice
filters. If a safe fix drags on for another year or two -- either because
it's just too hard or because nobody quite has the time -- is that just the
way it goes? I don't know. It's not obvious to me that breaking
compatibility with Enketo and relying on current behavior isn't the
welfare-maximizing choice.

In the end, I believe that Mitch's suggestion re: compatibility is maybe
necessary (storing in the XML some indication of how current() in choice
filters should be interpreted). And if we totally screwed up the fix and
broke a bunch of use cases, at least it might offer some escape for
affected users (if we allowed an override on the settings sheet).

So no, I don't agree that it's as simple as "fix the bug and make people
upgrade."

Chris

On Tue, Dec 1, 2015 at 5:11 PM Martijn van de Rijdt mar...@enketo.org wrote:

I can't help thinking that we're overthinking this
backwards-compatibility issue.

Ideally we have old clients that have the bug and new clients that do
not have the bug. Isn't that the case with any other major bug fix? If an
old form stops working in the new version of ODK Collect, the survey owner
will either have to upload a new version of the form or the user will have
to downgrade ODK Collect. Surely this will be a problem for some users, but
for how many? A small fraction of one percent seems like a reasonable guess.

Maybe the backwards-compatibility should be addressed just by warning
users, as Yaw suggested way above.

Any 'side-effects' of fixing the bug, like with creating itemsets, can
be tested for, I presume, before deploying the fix.

One way to detect at least a significant portion of the current
current() users, is to do a regex search for any usage of the
'current()/path/to/node' where current()/ is not followed by '../'. That
usage is specific to this bug to refer to a sibling node and will not occur
any more after the bug is fixed (because a question cannot have children).

On Mon, Nov 30, 2015 at 11:43 AM, Christopher Robert < cro...@surveycto.com> wrote:

I wasn't going to suggest adding the different parameters to any of
the XLSForm templates -- so they wouldn't become cluttered with this stuff.
Really, it would just be an emergency option for somebody requiring
different compatibility for some component that was out of sync with their
other components. Ideally, nobody would actually have to use the settings
sheet to override any of these options.

To be specific, my proposal would be that everybody
support current_always_field=false today, and that this be how pyxform
today defaults. Ideally, this would include Enketo, as Collect and
SurveyCTO's Web Collect (and anything else using the ODK JavaRosa) already
support current_always_field=false as their current behavior. Then, we
figure out how to get current_always_field=true to work in JavaRosa and
phase that in, eventually changing the pyxform default to that once we
think that enough components support it.

We have thousands of devices out there using older Collect versions,
even well after their servers have been updated... so my guess is that it's
some of those users who will end up having to use the settings-sheet option
to force current_always_field=false after we've changed the default to true.

Does all of that make sense?

Chris

On Sun, Nov 29, 2015 at 6:48 PM Yaw Anokwa yan...@nafundi.com wrote:

All this sounds fine to me, but wanted to raise some potential issues.
Discrete parameters also means folks can pick and choose what behavior
they want and it's hard to move the entire community forward that way.
Also, once you get past two or three, you end up with a settings sheet
of lots of flags to toggle and that isn't a great user experience.

Yaw

Need ODK consultants? Nafundi provides form design, server setup,
in-field training, and software development for ODK. Go to
https://nafundi.com to get started.

On Sun, Nov 29, 2015 at 6:47 PM, Christopher Robert cro...@surveycto.com wrote:

Having a number of discrete parameters that are overridable on the
settings
sheet makes sense to me (like "current_references_parent" or perhaps
"current_always_field"). We could then add model parameters and
change the
pyxform default over time, but users could always override on their
settings
sheets as needed for specific compatibility.

For example, our SurveyCTO pyxform for 2.10 (we've given up on 2.01
now)
could default to current_always_field=false, which would mean that
JR would
be backward-compatible and pyxform would emit relative paths for the
choice_filter column assuming that current() references the parent.
This
would break all choice_filters in Enketo, but our Enketo users
could set
current_always_field=true in their settings sheets.

For that to work, though, we'd also have to add support for
current_always_field=true in JR. We could do that as Meletis has
proposed,
which may have other side effects -- we'll have to see (e.g., for
itemset
copying).

Alternatively, we could try to convince Martijn to support
current_always_field=false, which is likely trivial. If he would do
that,
then we could push JR's support for current_always_field=true to
when we
could figure out how to do it safely. Also, it would mean producing
a
version of Enketo that supports relative paths in choice filters in
a manner
compatible with ODK Collect. Ona, Kobo, and other Enketo users could
rejoice, able to produce forms that use current() (like those that
reference
same-repeat values) in their choice_filters. That boon to users has
to be
worth something, even if a principle is slightly violated
(supporting
something outside the XPath spec).

Chris

On Tue, Nov 24, 2015 at 8:43 PM Alex Dorey dore...@gmail.com wrote:

Hi, Just catching up on the discussion. Sorry if I'm a bit behind
in my
understanding of things.

Regarding the pyxform changes--

  1. Instead of the date, could we have the configuration parameter
    describe
    the specific change?

So instead of:

revision="2015-11-24"

we specify something like:

current_references_parent=True (default to False with the next
pyxform
version bump)

  1. Could we put this configuration into the xlsform's settings
    sheet? I
    think this would be easier to test and configure. We could also
    add a way to
    override the setting with an extra argument to the CLI or xls2json
    call.

On Tuesday, November 24, 2015 at 3:26:18 PM UTC-5, Mitch wrote:

I agree. Having just delved into the itemset code, there are a
number of
issues that would need to be cleaned up.

What I would propose is to leave JR / ODK Collect broken as-is.
This
will be future work.

The python code will then be the focus of the tweaks.

(1) change:

json_survey =

xls2json.workbook_to_json(json.loads(request.POST['workbookJson']),

form_name=form_name, warnings=warningsList)

to add an optional revision argument:

json_survey =

xls2json.workbook_to_json(json.loads(request.POST['workbookJson']),

form_name=form_name, warnings=warningsList, revision='2015-11-24')

(2) change that implementation to:

(2a) store the revision string somewhere in the returned json
object.

(2b) expand ${fieldname} with the new relative path expansion
only if the
passed-in revision is >= '2015-11-24' ; otherwise, use the
absolute path
expansion.

(3) change:

create_survey_element_from_dict()

to emit the revision value as an attribute on the element.

===============
The JR itemset bug will be fixed at a later date.

When that is fixed, we can introduce a new revision string for
that
behavior change.

i.e., once we have the revision string moving through the system,
we can
begin to support non-backward-compatible changes to behaviors in
XLSForm and
JR form processing.

For the time being, the ODK XLSForm converter will not supply a
revision
argument, and not make use of this pyxform enhancement. Once the
JR fix is
done, we will then support that new revision and the current
behavior,
defaulting to the new behavior, but supporting users in
generating the old
behavior.

Mitch

On Tue, Nov 24, 2015 at 8:44 AM, Martijn van de Rijdt < mar...@enketo.org> wrote:

I appreciate the creative solution. It does sound like it will
become a
bit of a headache to manage and explain to users though. Whether
it's worth
it depends on how many active forms are out there that rely on
the bug. We
should get Ona's and KoBo's input on this too, I think.

From Enketo's side, relative paths were never supported in
choice-filters (to my embarrassment and surprise - which led to
my
question). As a result, there is no significant issue with
backwards
compatibility in Enketo once it is fixed in JR. Not sure how
meaningful this
is to ODK Collect, but no Enketo user has ever reported this
lack of
support. I think to date it's been too hard to craft such
choice-filters.
Because of the way Enketo is integrated into KoBo, Ona, Formhub,
I'd be
surprised if there were a significant amount of those users
using relative
choice-filters.

I do indeed feel strongly about following the XForms 2.0 spec
here. I
think relative path support is going to be very useful in the
future, and I
agree that we need to look at the bigger picture and do this
correctly, so
it won't catch up with us.

On Tue, Nov 24, 2015 at 4:24 AM, Christopher Robert cro...@surveycto.com wrote:

Okay, well, this is turning into a bigger and bigger project
and we may
well miss the window for our 2.01 maintenance release (and/or
make the
changes so extensive and scary that we cannot in good
conscience include
them in a maintenance release).

However, your idea for making JR sensitive to the revision value
emitted by pyxform opens a new possibility: we could release
the pyxform
change with 2.01 so that users can begin to benefit from the
relative paths
(relying on today's current() behavior), then assume that when
JR is "fixed"
it will only be fixed for XML emitted by newer pyxforms that
would have
updated their paths to account for the fix (using your
revision-value idea).
The primary drawback to this approach is that we would
basically break all
cascading-select filter expressions for SurveyCTO+Enketo users
(since
Martijn clearly wants to implement the "fixed" version of
current() rather
than the JavaRosa-compatible version of current()).

Meletis also found another issue relating to the evaluation
context in
jr:choice-name() (
https://github.com/opendatakit/opendatakit/issues/1180),
so we have that to contend with as well. If we can't safely
resolve the
context for jr:choice-name(), then that would present another
major drawback
and further lessen our enthusiasm for this entire adventure.

Best,

Chris

On Mon, Nov 23, 2015 at 5:45 PM Mitch Sundt < mitche...@gmail.com> wrote:

And, if it isn't clear, to preserve backward-compatible
operations,
XLSForm should not expand ${fieldname} with a relative path
unless the
revision value is the newer one; if the revision is not
specified, it would
emit absolute paths like it currently does.

I also switched to use a date value for the revision value
("2015-11-23") which allows simple lexical comparisons.

On Mon, Nov 23, 2015 at 1:03 PM, Mitch Sundt < mitche...@gmail.com> wrote:

w.r.t. backward compatibility

I too am concerned.

A proposed solution:

  1. Add a command line argument or other mechanism to request
    that
    XLSForm inject a 'revision="2015-11-23"' attribute into the
    element.
    e.g.,

This is used to alter the behavior of the XForms engine to
accommodate backward-compatible bug behaviors.

We can't identify the bugs (e.g, 'legacy="itemset-parent"'),
since
that would break older forms run on the new ODK Collect
releases.
We have to advance the revision code as bugs are fixed.

I think an integer value is fine for this.

I don't anticipate making a lot of revisions, and I think the
revisions would be sequentially applied (you can't
pick-and-choose what bugs
you include).

In the future, this would allow us to re-do the date-time
behavior.

  1. Modify the XLSForm tools (e.g., the online form generator)
    to
    enable the user to select what revision they want to emit
    (much like we did
    last year with the JR form eval choice setting within ODK
    Collect). We can
    default to emitting the latest revision if the user does not
    make a choice.

  2. Augment JR to look for this revision attribute and
    store/restore
    it in the FormDef binary object.

Based upon this value, alter the behavior of the JR itemset
evaluation to fix the path bug.

  1. Modify ODK Validate to scan for syntax that may be
    impacted by a
    revision (to advise whether to change something). Not sure
    how complex that
    might be. May not be always possible?

w.r.t. impacts of change

Thank you Meletis for researching when this change occurred.

I will have to ponder what effect the parent/node change
would have
on itemset copying. Hadn't thought that there was any
relationship between
the two, but perhaps there is.

Yaw also reported that the Cascading Select itemsets on
opendatakit.appspot.com are not working with the 1.4.7 rev
1053 release (and
they work but give odd 'null' resolutions on 1057). This is
in a related
area of this same code, so I will hopefully gain a better
understanding of
dynamic itemsets as I uncover the causes of that problem.


Mitch

On Mon, Nov 23, 2015 at 5:59 AM, Christopher Robert cro...@surveycto.com wrote:

So just to be clear, the "fix" we're testing here reverts
the entire
context within that populateDynamicChoices() function, to be
the field
instead of the parent -- so the effects are potentially much
wider than just
what current() returns. Potentially anything and everything
that depends on
context can change now, since the context will shift from
parent to field.

This means that we revert a change from 5 years ago and
alter the
context from now on. Neither Meletis nor I feel terribly
confident that we
can entirely predict the consequences to backward
compatibility. We're
testing the things we can think to test, but honestly it's
XML users in the
broader ODK community who face the greatest risks here;
SurveyCTO users by
and large use XLSForm and thus the potential XPath syntax
consequences are
sharply limited.

Are you guys very sure that you want to take this path? I
can tell
you now that if we make this change and then there is some
outcry after the
next ODK Collect release, SurveyCTO Collect will very likely
stick with the
new change (since we will have changed pyxform and we'll
then have too many
backward-compatibility concerns to then reverse the change).

Thanks,

Chris

On Mon, Nov 23, 2015 at 5:23 AM mel...@surveycto.com wrote:

Just for the history's sake, I found the time where the
behavior
changed. It was at March 11, 2010:

https://bitbucket.org/m.sundt/javarosa/commits/b41875b2c23c9a21a868a5d2dc0ffc1f6b5b0b65#Lcore/src/org/javarosa/core/model/FormDef.javaT635

Take a look only in the changes of the
"populateDynamicChoices"
method. The context reference used to be the "current"
field, but after the
changes above it started be the field's parent (by adding
getParent()).
Perhaps there was a reason for that change, but perhaps
not, but I think all
of us agree that this should be fixed, even after 5 and a
half years.

Meletis

On Monday, November 23, 2015 at 12:08:47 PM UTC+2, mel...@surveycto.com wrote:

Hi Mitch,

Please take a look at the code comment in line 31 here:

https://bitbucket.org/m.sundt/javarosa/src/bf17fb749b74e53915f086ca3827c3c6fe73fd07/core/src/org/javarosa/core/model/ItemsetBinding.java?at=default&fileviewer=file-view-default#ItemsetBinding.java-31

It clearly says: "ref of the control parent
(group/formdef) of
itemset question"

So, for some reason, they wanted it to point to the
parent. I also
see that when the dynamic choices are populated, the
evaluation context is
wrapped around that parent (contextRef object), that's why
current() refers
to the parent group when evaluating itemsets nodesets.

I will run a few tests by changing the contextRef field
value, I
just wanted to mention that in case you remember a reason
for that code
comment.

Meletis

On Friday, November 20, 2015 at 8:49:12 PM UTC+2, Mitch wrote:

Yes, I think it should be a one-line change; just don't
know
where.

Mitch

(I do jar-updates over Christmas / New-Years. Tedious,
but
doesn't require a whole lot of attention.)

On Fri, Nov 20, 2015 at 10:36 AM, mel...@surveycto.com wrote:

Hi Mitch,

Thank you for all the details, I guess I was missing
some of the
context of the original issue.

We will give it a shot to see if we can come up with a
few ideas
how to fix the evaluation content of the itemset
nodeset. I just hope it
will not be like last year with the "form processing
logic" in JR, otherwise
I believe that every Christmas will be a JR Christmas. :slight_smile:

Meletis

On Friday, November 20, 2015 at 7:29:14 PM UTC+2, Mitch wrote:

The form was attached in one of my earlier replies.

On Fri, Nov 20, 2015 at 9:27 AM, Mitch Sundt mitche...@gmail.com wrote:

And you see this in constraint expressions where you
write:

. > 6

I.e., current()/. should refer to the field, and not
the group
containing the field, as it currently does when
evaluating expressions in
the itemset nodeset.

On Fri, Nov 20, 2015 at 9:26 AM, Mitch Sundt mitche...@gmail.com wrote:

The issue is that there is a discrepancy in the
meaning of
'.' across the different contexts of: calculate,
relevant, , and
these itemset nodeset expressions.

In other usages, you need to reference this other
field with
current()/../filterfield ; but in the nodeset
expression, you would
reference it with current()/./filterfield

I.e., the notion of '.' is different when used within
a
nodeset expression and when used within a constraint
expression or an
field.

See the attached NBiggestOfSet form. For the label,
it has:

    <select1

ref="/NBiggestOfSet/education/edu_level">

      <label>What is <output

value="current()/../referred_attendee_name"/>'s level
of education?
...

For a constraint, it has:
<bind

calculate="/NBiggestOfSet/attendee/name[position(..)=int(current()/../referred_attendee)]"

nodeset="/NBiggestOfSet/education/referred_attendee_name" readonly="true()"

type="string"/>

These each use the current()/../ to reference the
group (
/NBiggesetOfSet/education ).

But, within the nodeset definition (if this form used
one),
you would need to use
current()/./referred_attendee_name to reference a
field in that group.

On Fri, Nov 20, 2015 at 8:22 AM, < mel...@surveycto.com> wrote:

Hi Mitch,

I think I'm missing something but I think that
nothing is
broken and that the only issue is that the expression
current()/../filterfield in your example above
should be just changed to
current()/filterfield. If you make that change,
everything should work
because it will be resolved to:

/data/group1/filterfield

Why should a "../" be present there? We have used
current()/somefield successfully in one of our test
forms and it works fine.
And I think it would also work great in your
example. Wouldn't it? Or am I
totally out of context?

Meletis

On Friday, November 20, 2015 at 5:35:16 PM UTC+2, Mitch wrote:

Hi Meletis,

That's not the issue --

The issue is that if you use a filter condition in
the
&g

--
Revolutionizing data collection since 2012.

Enketo https://enketo.org/ | LinkedIn
http://www.linkedin.com/company/enketo-llc | GitHub
https://github.com/enketo | Twitter https://twitter.com/enketo
| Blog http://blog.enketo.org/

--
You received this message because you are subscribed to the Google Groups
"ODK Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to opendatakit-developers+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--
Mitch Sundt
Software Engineer
University of Washington
mitchellsundt@gmail.com

--
You received this message because you are subscribed to the Google Groups
"ODK Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to opendatakit-developers+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

It isn't that dire.

Not only do you need to then update 100% of your devices with a high
degree of certainty, but you also need to re-generate the XML for every
form on your server. And not even that is enough. Then you need to update
every form definition on every device.

··· ================= *Individual Device Safeguards*

Yes, all the devices need to be updated. And yes, this is the biggest pain
point. We CAN change the form download task so that it first scans the
form to detect unsupported revision tags, and refuses to install the form
definition on the device. And then throws up a Dialog warning the user of
the inappropriate form and prodding them to update ODK Collect. This would
preserve the older version of that individual form on that one device.
Survey CTO already added functionality to the download task to ensure that
all form attachments were downloaded before the form was copied into the
/sdcard/odk/forms directory and became 'live'. So this would be an extra
validity check before that wholesale move, allowing your non-upgraded
devices to stay 'in the past' w.r.t. form definitions they already have
until you upgrade the device.

This prevents catastrophic failure of an individual device, but does not
ensure complete migration of all the devices to the latest ODK Collect.

================
Device Fleet Management

ODK Aggregate, at least, does not have any mechanism to track all devices
and what their ODK Collect versions are. This is something that ODK Manage
was intended to do, but that never got implemented.

I haven't looked, but there are undoubtedly 3rd party tools that can manage
fleets of devices and communicate their configurations back to a central
location, and force-push updates onto individual devices. All the
anti-theft phone-lockout tools are basically doing that.

================
Form Revision Management

You do not need to upgrade all the forms at once.
Because the revision flags are in each form, you can keep running the forms
that require the bugs.
Even as you update your XLSForm, server and ODK Collect.

Depending upon how end-to-end your solution is, when running XLSForm to
update an existing form_id, you could access your live data collection
server to retrieve the existing form_id, and then, by default, apply the
same revision tags when making updates to that form_id. And only change
those revision tags through explicit user action.

That would allow users to easily edit the form to add another language or
fix a typo without much thought. It could also be expanded to run the
verification of identical data-models at that time (rather than later when
you try to upload the revision to the server) and to confirm that the form
version is appropriately larger than the existing form.

On Wed, Dec 2, 2015 at 4:37 PM, Christopher Robert crobert@surveycto.com wrote:

Thanks all, I appreciate all of the thinking going into this.

At the risk of thoroughly exhausting everybody's patience, I'm still not
sure that we have a strategy that can be described as reasonable from an
existing user's point of view. Let me explain why.

Say we release SurveyCTO 2.10 with three changes:

  1. We add the fix to JR and have it apply whenever the appropriate flag is
    set in the XForm model.

  2. We default all pyxform generation (which is basically invisible to the
    user in SurveyCTO) to include the appropriate flag so that newly-generated
    XML will be interpreted with the fixed JR.

  3. We default pyxform to output XPaths in choice-filter expressions using
    current() so that within-context repeat references finally work. This code,
    by default, relies on the fix (so paths are relative to the field, not the
    group). (It also changes ALL XPath expressions, not just those referencing
    within-repeat values, because we're not clever enough to be able to
    distinguish the different cases in that python code.)

Now say you're an upgrading user and you upgraded because SurveyCTO 2.10
also included a beautiful and astonishingly powerful web-based form
designer. You upgraded to get this new designer and you love it and now you
upload a new or revised form. That form happens to have a choice-filter
expression. Now here's where things get ugly...

That form won't work on your SurveyCTO 2.01, 2.00, 1.402, etc. devices.
Probably there's some nasty XPath parsing error when you get to where the
choice filter kicks in, maybe it even crashes your Collect. That form might
have worked the day before, and maybe all you changed was fixing a typo in
a label, but now suddenly it doesn't work on all of these devices.

So here's where we say, okay, the user has two choices: (1) upgrade all
devices to the 2.10 Collect, or (2) find and set the "stay in the past"
option. The second option is good in principle -- because it provides a
kind of quick save -- but then what does it take to ever migrate away from
that "stay in the past" setting? Not only do you need to then update 100%
of your devices with a high degree of certainty, but you also need to
re-generate the XML for every form on your server. And not even that is
enough. Then you need to update every form definition on every device.

And if you miss a few devices and leave them on the old version, or miss a
few form definitions here or there? Cryptic errors, inability to finalize
forms, confusion, anger, frantic calls to support, etc., etc.

You don't need to be a human-centered design expert to see that this is
kind of a disaster from a user-experience point of view. And, for us, this
would be multiplied across hundreds of migrating users, essentially eating
a lot of our 2016 and causing a tremendous amount of ill will.

The only upgrade path for existing users that makes any sense is one where
we basically emit the "stay in the past" flag always for everyone. You can
all turn your noses up at the idea of indefinite reliance on what was
initially a bug, and I get it. But at least here at SurveyCTO, we're not
going to put our users through the kind of migration path described above.

If relying on the bug is just too unpalatable, then ultimately I don't
think that we can change pyxform to emit XPaths using current(). It's just
too fraught and we don't have a good way to manage the upgrade path without
torturing existing users. So fine, our awesome pyxform fix to magically
make within-repeat references start working is just not destined to see the
light of day. Rather, we need to make pyxform smart enough to only use
current() for paths to repeated values (which never worked in old JR's
anyway). Or we need to just provide a new solution for referencing repeated
values without current() (perhaps a new custom JR function, which everybody
loves).

And in that case, sure, I'm much more relaxed about the strategies for
upgrading users since very few are likely to be using current(). I won't
want to overly torture those users, but I won't lose a ton of sleep over
them since even users who use current() now are unlikely to use it in a lot
of forms.

I'll see if we can figure out how to make the pyxform change in such a way
that it only changes the XPaths for fields within repeat groups. But I
might also see if we can instead find a totally different way to solve or
work around this problem. Or, I'll see how I feel about pushing this off
for another year. Fixing choice filters to be able to work will within
repeat groups was not meant to be a huge project.

Thanks again,

Chris

On Wed, Dec 2, 2015 at 2:48 PM Mitch Sundt mitchellsundt@gmail.com wrote:

w.r.t. moving forward...

Even if this particular change impacts a very small number of forms, we
should figure out how to implement non-backward-compatible changes so that
they are safe moving forward. We have no visibility into who uses ODK
Collect and ODK Aggregate and no reliable way to notify them of changes.

This could be a huge deal when or if we ever revise the JR engine to
properly handle timezones, dates and times, and date and time arithmetic.

Or if we find a need to change XLSForm to emit XForms features or XPath
functions that currently don't work (e.g., to use [1] or position(), or
XForms events, for example).

Note that these non-backward-compatible changes extend to not just
XLSForm and ODK Collect, but also to servers. You can use the same revision
tag to indicate new or different processing to be taken on the server.

=============
Managing this change requires:

I believe that this requires:

(1) the XML form definition is the place to define the
feature-enabling-labels that the form requires of the device and server
processing steps.

(2) when the JR engine or any tool implements non-backward-compatible
changes, it only activates them when it detects the presence of those
feature-enabling-label in the XML form definition. Omitting that
feature-enabling-label is a directive to "stay in the past".

(3) ODK Collect and other tools processing the XML form definition or the
XML submission documents should warn the user (e.g., pop up a Toast) if
they are running a version that does not recognize the
feature-enabling-labels in the form's revision tag (i.e., they are given a
form that indicates an advanced feature that they are unaware of) -- this
notification informs the user that they should upgrade that tool to
correctly process the directive. Alternatively, the tool could reject the
form, Force Close or abort, which is the current behavior when ODK
Aggregate or ODK Collect gets an XForms definition that uses a new,
unrecognized, function, etc. ;-<

=============
The stable upgrade path is to:

(1) update your form distribution and data aggregation server(s),
(2) update your collection devices (or collection processing server --
e.g., Enketo), and
(3) only then update the form generation tool to emit forms with the
non-backward-compatible change.

=============
Specifying the Revision settings:

I believe the XLS file should not contain a setting to select or
specify the revision
.

I believe this should be specified at the time the XLSForm converter is
executed on the source XLS file.
I.e., the web UI for these tools should have a spinner or other UI
element that enables users to choose to "stay in the past"

This forces users to actively stay in the past.

The ODK site will probably achieve this with a drop-down that allows the
user to specify, e.g.,

"full current functionality"
"generate a form definition compatible with ODK Collect 1.4.7 and earlier"

To allow all groups to independently advance the functionality within
their tool families, the combination of features that each of these
drop-down choices corresponds to should be documented by those groups.

I.e., I see a future where a form generated on Ona for Enketo may not be
compatible with stock ODK Collect unless an appropriate compatibility
drop-down is selected.

This could allow, for example, Survey CTO to implement and recoup costs
for JR date-time fixes or other XForms extensions within their ecosystem
before releasing that fix back to the wider community.

I do not see any value in offering a user the choice of which features
are enabled or disabled. I suppose some sites could offer that, but it
seems overly complex. Users really just need a way to specify compatibility
to whatever older version of the tools they are using.

==============
XLSForm functionality

I am fine with the XLSForm generator taking a comma-separated list of
features to enable, and passing that list through via the revision tag to
the the XML form definition.

Upon reflection, I think this is the only way to manage multiple
independent parallel contributions to the various codebases. It allows
different groups to add and enable different functionalities at different
rates.

We just need to maintain a global list of all of the feature labels that
every group is using so that we don't choose names that collide.

And, some of these feature labels may simply pass through the XLSForm
generator and on to the downstream tools. e.g., date-time fixes might not
have any impact on XLSForm, but would alter how the JR engine processed the
form.

On Wed, Dec 2, 2015 at 10:25 AM, Martijn van de Rijdt <martijn@enketo.org wrote:

Ok, I was just wondering if it's really worth the effort. KoBo found 3
forms using current(). Not sure how many are actively used still. For them
it would seem to me easier to just contact that/those user(s) individually
to come up with a plan.

I can live with the XForm attribute and XLSForm setting addition.

Chris, to respond to your question about adding support for the bug to
Enketo. I don't currently see a reason to do so, as no Enketo user has ever
relied on the bug and I highly doubt any will in the future. However, I
don't think this should be relevant to this discussion. We're all in
agreement that we want the community to move forward with the correct
syntax.

On Tuesday, December 1, 2015 at 4:58:45 PM UTC-7, Christopher Robert wrote:

We have individual users with 500+ devices in the field. Updating
devices -- or even keeping track of which devices are on which versions --
is a major undertaking. In total, we're talking many thousands of devices
spread across many countries. The problem is not a small one, and our
commitment to backward compatibility is one that our users not only
appreciate but expect.

ODK/SurveyCTO Collect continues to use older versions of forms to view,
edit, or finalize submissions begun under those older versions, even when
newer versions exist. So the backward-compatibility challenge is a
formidable one. If you change the way JavaRosa interprets expressions in
the kind of blanket way that you suggest, then users will be faced with the
impossible choice of running software that fails to function properly with
existing, in-progress, or old submissions, or running software that fails
to function properly with newer submissions. Multiply the difficulty of the
choice and the potential confusion of XPath parsing errors times
however-many devices there are in the field.

No, this is a legitimately difficult situation that requires a
tremendous amount of care. Changing how XPath expressions are interpreted
is a big deal for the Collect world, though I can see why it would be
considerably less so in the Enketo world.

Now, I will admit that there are bound to be very few users who already
use current() in choice filters. Mitch suggested the approach a few times
in the public forum, but even I failed to get it to work the first several
times I tried to follow his advice. So chances are that not many projects
are using current() today. (I know of 1-2, maybe there are 8-10 total.
That's just a guess.)

Enketo aside, we could -- today! -- allow people to reference
in-context repeated values by exploiting the current JR support for
current(). For some users/projects, there would be an immediate benefit.
But obviously that relies on "the bug" and so it would expand the potential
need for backward compatibility. And it would break choice-filter
compatibility between Collect and Enketo (say, for SurveyCTO users at
least).

It's easy to say "just fix the bug and then this problem goes away,"
but the fact is that we don't quite know how to fix it without breaking
other things; we have a candidate for a fix, but it's seriously scary. It
may take time. And meanwhile, we have already gone multiple years with
users waiting to be able to reference within-context repeated values in
choice filters. If a safe fix drags on for another year or two -- either
because it's just too hard or because nobody quite has the time -- is that
just the way it goes? I don't know. It's not obvious to me that breaking
compatibility with Enketo and relying on current behavior isn't the
welfare-maximizing choice.

In the end, I believe that Mitch's suggestion re: compatibility is
maybe necessary (storing in the XML some indication of how current() in
choice filters should be interpreted). And if we totally screwed up the fix
and broke a bunch of use cases, at least it might offer some escape for
affected users (if we allowed an override on the settings sheet).

So no, I don't agree that it's as simple as "fix the bug and make
people upgrade."

Chris

On Tue, Dec 1, 2015 at 5:11 PM Martijn van de Rijdt mar...@enketo.org wrote:

I can't help thinking that we're overthinking this
backwards-compatibility issue.

Ideally we have old clients that have the bug and new clients that do
not have the bug. Isn't that the case with any other major bug fix? If an
old form stops working in the new version of ODK Collect, the survey owner
will either have to upload a new version of the form or the user will have
to downgrade ODK Collect. Surely this will be a problem for some users, but
for how many? A small fraction of one percent seems like a reasonable guess.

Maybe the backwards-compatibility should be addressed just by warning
users, as Yaw suggested way above.

Any 'side-effects' of fixing the bug, like with creating itemsets, can
be tested for, I presume, before deploying the fix.

One way to detect at least a significant portion of the current
current() users, is to do a regex search for any usage of the
'current()/path/to/node' where current()/ is not followed by '../'. That
usage is specific to this bug to refer to a sibling node and will not occur
any more after the bug is fixed (because a question cannot have children).

On Mon, Nov 30, 2015 at 11:43 AM, Christopher Robert < cro...@surveycto.com> wrote:

I wasn't going to suggest adding the different parameters to any of
the XLSForm templates -- so they wouldn't become cluttered with this stuff.
Really, it would just be an emergency option for somebody requiring
different compatibility for some component that was out of sync with their
other components. Ideally, nobody would actually have to use the settings
sheet to override any of these options.

To be specific, my proposal would be that everybody
support current_always_field=false today, and that this be how pyxform
today defaults. Ideally, this would include Enketo, as Collect and
SurveyCTO's Web Collect (and anything else using the ODK JavaRosa) already
support current_always_field=false as their current behavior. Then, we
figure out how to get current_always_field=true to work in JavaRosa and
phase that in, eventually changing the pyxform default to that once we
think that enough components support it.

We have thousands of devices out there using older Collect versions,
even well after their servers have been updated... so my guess is that it's
some of those users who will end up having to use the settings-sheet option
to force current_always_field=false after we've changed the default to true.

Does all of that make sense?

Chris

On Sun, Nov 29, 2015 at 6:48 PM Yaw Anokwa yan...@nafundi.com wrote:

All this sounds fine to me, but wanted to raise some potential
issues.
Discrete parameters also means folks can pick and choose what
behavior
they want and it's hard to move the entire community forward that
way.
Also, once you get past two or three, you end up with a settings
sheet
of lots of flags to toggle and that isn't a great user experience.

Yaw

Need ODK consultants? Nafundi provides form design, server setup,
in-field training, and software development for ODK. Go to
https://nafundi.com to get started.

On Sun, Nov 29, 2015 at 6:47 PM, Christopher Robert cro...@surveycto.com wrote:

Having a number of discrete parameters that are overridable on the
settings
sheet makes sense to me (like "current_references_parent" or
perhaps
"current_always_field"). We could then add model parameters and
change the
pyxform default over time, but users could always override on
their settings
sheets as needed for specific compatibility.

For example, our SurveyCTO pyxform for 2.10 (we've given up on
2.01 now)
could default to current_always_field=false, which would mean that
JR would
be backward-compatible and pyxform would emit relative paths for
the
choice_filter column assuming that current() references the
parent. This
would break all choice_filters in Enketo, but our Enketo users
could set
current_always_field=true in their settings sheets.

For that to work, though, we'd also have to add support for
current_always_field=true in JR. We could do that as Meletis has
proposed,
which may have other side effects -- we'll have to see (e.g., for
itemset
copying).

Alternatively, we could try to convince Martijn to support
current_always_field=false, which is likely trivial. If he would
do that,
then we could push JR's support for current_always_field=true to
when we
could figure out how to do it safely. Also, it would mean
producing a
version of Enketo that supports relative paths in choice filters
in a manner
compatible with ODK Collect. Ona, Kobo, and other Enketo users
could
rejoice, able to produce forms that use current() (like those that
reference
same-repeat values) in their choice_filters. That boon to users
has to be
worth something, even if a principle is slightly violated
(supporting
something outside the XPath spec).

Chris

On Tue, Nov 24, 2015 at 8:43 PM Alex Dorey dore...@gmail.com wrote:

Hi, Just catching up on the discussion. Sorry if I'm a bit behind
in my
understanding of things.

Regarding the pyxform changes--

  1. Instead of the date, could we have the configuration parameter
    describe
    the specific change?

So instead of:

revision="2015-11-24"

we specify something like:

current_references_parent=True (default to False with the next
pyxform
version bump)

  1. Could we put this configuration into the xlsform's settings
    sheet? I
    think this would be easier to test and configure. We could also
    add a way to
    override the setting with an extra argument to the CLI or
    xls2json call.

On Tuesday, November 24, 2015 at 3:26:18 PM UTC-5, Mitch wrote:

I agree. Having just delved into the itemset code, there are a
number of
issues that would need to be cleaned up.

What I would propose is to leave JR / ODK Collect broken as-is.
This
will be future work.

The python code will then be the focus of the tweaks.

(1) change:

json_survey =

xls2json.workbook_to_json(json.loads(request.POST['workbookJson']),

form_name=form_name, warnings=warningsList)

to add an optional revision argument:

json_survey =

xls2json.workbook_to_json(json.loads(request.POST['workbookJson']),

form_name=form_name, warnings=warningsList,
revision='2015-11-24')

(2) change that implementation to:

(2a) store the revision string somewhere in the returned json
object.

(2b) expand ${fieldname} with the new relative path expansion
only if the
passed-in revision is >= '2015-11-24' ; otherwise, use the
absolute path
expansion.

(3) change:

create_survey_element_from_dict()

to emit the revision value as an attribute on the
element.

===============
The JR itemset bug will be fixed at a later date.

When that is fixed, we can introduce a new revision string for
that
behavior change.

i.e., once we have the revision string moving through the
system, we can
begin to support non-backward-compatible changes to behaviors in
XLSForm and
JR form processing.

For the time being, the ODK XLSForm converter will not supply a
revision
argument, and not make use of this pyxform enhancement. Once the
JR fix is
done, we will then support that new revision and the current
behavior,
defaulting to the new behavior, but supporting users in
generating the old
behavior.

Mitch

On Tue, Nov 24, 2015 at 8:44 AM, Martijn van de Rijdt < mar...@enketo.org> wrote:

I appreciate the creative solution. It does sound like it will
become a
bit of a headache to manage and explain to users though.
Whether it's worth
it depends on how many active forms are out there that rely on
the bug. We
should get Ona's and KoBo's input on this too, I think.

From Enketo's side, relative paths were never supported in
choice-filters (to my embarrassment and surprise - which led to
my
question). As a result, there is no significant issue with
backwards
compatibility in Enketo once it is fixed in JR. Not sure how
meaningful this
is to ODK Collect, but no Enketo user has ever reported this
lack of
support. I think to date it's been too hard to craft such
choice-filters.
Because of the way Enketo is integrated into KoBo, Ona,
Formhub, I'd be
surprised if there were a significant amount of those users
using relative
choice-filters.

I do indeed feel strongly about following the XForms 2.0 spec
here. I
think relative path support is going to be very useful in the
future, and I
agree that we need to look at the bigger picture and do this
correctly, so
it won't catch up with us.

On Tue, Nov 24, 2015 at 4:24 AM, Christopher Robert cro...@surveycto.com wrote:

Okay, well, this is turning into a bigger and bigger project
and we may
well miss the window for our 2.01 maintenance release (and/or
make the
changes so extensive and scary that we cannot in good
conscience include
them in a maintenance release).

However, your idea for making JR sensitive to the revision
value
emitted by pyxform opens a new possibility: we could release
the pyxform
change with 2.01 so that users can begin to benefit from the
relative paths
(relying on today's current() behavior), then assume that when
JR is "fixed"
it will only be fixed for XML emitted by newer pyxforms that
would have
updated their paths to account for the fix (using your
revision-value idea).
The primary drawback to this approach is that we would
basically break all
cascading-select filter expressions for SurveyCTO+Enketo users
(since
Martijn clearly wants to implement the "fixed" version of
current() rather
than the JavaRosa-compatible version of current()).

Meletis also found another issue relating to the evaluation
context in
jr:choice-name() (
https://github.com/opendatakit/opendatakit/issues/1180),
so we have that to contend with as well. If we can't safely
resolve the
context for jr:choice-name(), then that would present another
major drawback
and further lessen our enthusiasm for this entire adventure.

Best,

Chris

On Mon, Nov 23, 2015 at 5:45 PM Mitch Sundt < mitche...@gmail.com> wrote:

And, if it isn't clear, to preserve backward-compatible
operations,
XLSForm should not expand ${fieldname} with a relative path
unless the
revision value is the newer one; if the revision is not
specified, it would
emit absolute paths like it currently does.

I also switched to use a date value for the revision value
("2015-11-23") which allows simple lexical comparisons.

On Mon, Nov 23, 2015 at 1:03 PM, Mitch Sundt < mitche...@gmail.com> wrote:

w.r.t. backward compatibility

I too am concerned.

A proposed solution:

  1. Add a command line argument or other mechanism to request
    that
    XLSForm inject a 'revision="2015-11-23"' attribute into the
    element.
    e.g.,

This is used to alter the behavior of the XForms engine to
accommodate backward-compatible bug behaviors.

We can't identify the bugs (e.g, 'legacy="itemset-parent"'),
since
that would break older forms run on the new ODK Collect
releases.
We have to advance the revision code as bugs are fixed.

I think an integer value is fine for this.

I don't anticipate making a lot of revisions, and I think the
revisions would be sequentially applied (you can't
pick-and-choose what bugs
you include).

In the future, this would allow us to re-do the date-time
behavior.

  1. Modify the XLSForm tools (e.g., the online form
    generator) to
    enable the user to select what revision they want to emit
    (much like we did
    last year with the JR form eval choice setting within ODK
    Collect). We can
    default to emitting the latest revision if the user does not
    make a choice.

  2. Augment JR to look for this revision attribute and
    store/restore
    it in the FormDef binary object.

Based upon this value, alter the behavior of the JR itemset
evaluation to fix the path bug.

  1. Modify ODK Validate to scan for syntax that may be
    impacted by a
    revision (to advise whether to change something). Not sure
    how complex that
    might be. May not be always possible?

w.r.t. impacts of change

Thank you Meletis for researching when this change occurred.

I will have to ponder what effect the parent/node change
would have
on itemset copying. Hadn't thought that there was any
relationship between
the two, but perhaps there is.

Yaw also reported that the Cascading Select itemsets on
opendatakit.appspot.com are not working with the 1.4.7 rev
1053 release (and
they work but give odd 'null' resolutions on 1057). This is
in a related
area of this same code, so I will hopefully gain a better
understanding of
dynamic itemsets as I uncover the causes of that problem.


Mitch

On Mon, Nov 23, 2015 at 5:59 AM, Christopher Robert cro...@surveycto.com wrote:

So just to be clear, the "fix" we're testing here reverts
the entire
context within that populateDynamicChoices() function, to
be the field
instead of the parent -- so the effects are potentially
much wider than just
what current() returns. Potentially anything and everything
that depends on
context can change now, since the context will shift from
parent to field.

This means that we revert a change from 5 years ago and
alter the
context from now on. Neither Meletis nor I feel terribly
confident that we
can entirely predict the consequences to backward
compatibility. We're
testing the things we can think to test, but honestly it's
XML users in the
broader ODK community who face the greatest risks here;
SurveyCTO users by
and large use XLSForm and thus the potential XPath syntax
consequences are
sharply limited.

Are you guys very sure that you want to take this path? I
can tell
you now that if we make this change and then there is some
outcry after the
next ODK Collect release, SurveyCTO Collect will very
likely stick with the
new change (since we will have changed pyxform and we'll
then have too many
backward-compatibility concerns to then reverse the change).

Thanks,

Chris

On Mon, Nov 23, 2015 at 5:23 AM mel...@surveycto.com wrote:

Just for the history's sake, I found the time where the
behavior
changed. It was at March 11, 2010:

https://bitbucket.org/m.sundt/javarosa/commits/b41875b2c23c9a21a868a5d2dc0ffc1f6b5b0b65#Lcore/src/org/javarosa/core/model/FormDef.javaT635

Take a look only in the changes of the
"populateDynamicChoices"
method. The context reference used to be the "current"
field, but after the
changes above it started be the field's parent (by adding
getParent()).
Perhaps there was a reason for that change, but perhaps
not, but I think all
of us agree that this should be fixed, even after 5 and a
half years.

Meletis

On Monday, November 23, 2015 at 12:08:47 PM UTC+2, mel...@surveycto.com wrote:

Hi Mitch,

Please take a look at the code comment in line 31 here:

https://bitbucket.org/m.sundt/javarosa/src/bf17fb749b74e53915f086ca3827c3c6fe73fd07/core/src/org/javarosa/core/model/ItemsetBinding.java?at=default&fileviewer=file-view-default#ItemsetBinding.java-31

It clearly says: "ref of the control parent
(group/formdef) of
itemset question"

So, for some reason, they wanted it to point to the
parent. I also
see that when the dynamic choices are populated, the
evaluation context is
wrapped around that parent (contextRef object), that's
why current() refers
to the parent group when evaluating itemsets nodesets.

I will run a few tests by changing the contextRef field
value, I
just wanted to mention that in case you remember a reason
for that code
comment.

Meletis

On Friday, November 20, 2015 at 8:49:12 PM UTC+2, Mitch wrote:

Yes, I think it should be a one-line change; just don't
know
where.

Mitch

(I do jar-updates over Christmas / New-Years. Tedious,
but
doesn't require a whole lot of attention.)

On Fri, Nov 20, 2015 at 10:36 AM, mel...@surveycto.com wrote:

Hi Mitch,

Thank you for all the details, I guess I was missing
some of the
context of the original issue.

We will give it a shot to see if we can come up with a
few ideas
how to fix the evaluation content of the itemset
nodeset. I just hope it
will not be like last year with the "form processing
logic" in JR, otherwise
I believe that every Christmas will be a JR Christmas.
:slight_smile:

Meletis

On Friday, November 20, 2015 at 7:29:14 PM UTC+2, Mitch wrote:

The form was attached in one of my earlier replies.

On Fri, Nov 20, 2015 at 9:27 AM, Mitch Sundt mitche...@gmail.com wrote:

And you see this in constraint expressions where you
write:

. > 6

I.e., current()/. should refer to the field, and not
the group
containing the field, as it currently does when
evaluating expressions in
the itemset nodeset.

On Fri, Nov 20, 2015 at 9:26 AM, Mitch Sundt mitche...@gmail.com wrote:

The issue is that there is a discrepancy in the
meaning of
'.' across the different contexts of: calculate,
relevant, , and
these itemset nodeset expressions.

In other usages, you need to reference this other
field with
current()/../filterfield ; but in the nodeset
expression, you would
reference it with current()/./filterfield

I.e., the notion of '.' is different when used
within a
nodeset expression and when used within a constraint
expression or an
field.

See the attached NBiggestOfSet form. For the label,
it has:

    <select1

ref="/NBiggestOfSet/education/edu_level">

      <label>What is <output

value="current()/../referred_attendee_name"/>'s
level of education?
...

For a constraint, it has:
<bind

calculate="/NBiggestOfSet/attendee/name[position(..)=int(current()/../referred_attendee)]"

nodeset="/NBiggestOfSet/education/referred_attendee_name" readonly="true()"

type="string"/>

These each use the current()/../ to reference the
group (
/NBiggesetOfSet/education ).

But, within the nodeset definition (if this form
used one),
you would need to use
current()/./referred_attendee_name to reference a
field in that group.

On Fri, Nov 20, 2015 at 8:22 AM, < mel...@surveycto.com> wrote:

Hi Mitch,

I think I'm missing something but I think that
nothing is
broken and that the only issue is that the
expression
current()/../filterfield in your example above
should be just changed to
current()/filterfield. If you make that change,
everything should work
because it will be resolved to:

/data/group1/filterfield

Why should a "../" be present there? We have used
current()/somefield successfully in one of our test
forms and it works fine.
And I think it would also work great in your
example. Wouldn't it? Or am I
totally out of context?

Meletis

On Friday, November 20, 2015 at 5:35:16 PM UTC+2, Mitch wrote:

Hi Meletis,

That's not the issue --

The issue is that if you use a filter condition in
the
&g

--
Revolutionizing data collection since 2012.

Enketo https://enketo.org/ | LinkedIn
http://www.linkedin.com/company/enketo-llc | GitHub
https://github.com/enketo | Twitter https://twitter.com/enketo
| Blog http://blog.enketo.org/

--
You received this message because you are subscribed to the Google
Groups "ODK Developers" group.
To unsubscribe from this group and stop receiving emails from it, send
an email to opendatakit-developers+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--
Mitch Sundt
Software Engineer
University of Washington
mitchellsundt@gmail.com

--
You received this message because you are subscribed to the Google Groups
"ODK Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to opendatakit-developers+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups
"ODK Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to opendatakit-developers+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--
Mitch Sundt
Software Engineer
University of Washington
mitchellsundt@gmail.com

Thanks, Mitch. Some great ideas in there.

Best,

Chris

··· On Thu, Dec 3, 2015 at 12:52 PM Mitch Sundt wrote:

It isn't that dire.

Not only do you need to then update 100% of your devices with a high
degree of certainty, but you also need to re-generate the XML for every
form on your server. And not even that is enough. Then you need to update
every form definition on every device.

=================
Individual Device Safeguards

Yes, all the devices need to be updated. And yes, this is the biggest pain
point. We CAN change the form download task so that it first scans the
form to detect unsupported revision tags, and refuses to install the form
definition on the device. And then throws up a Dialog warning the user of
the inappropriate form and prodding them to update ODK Collect. This would
preserve the older version of that individual form on that one device.
Survey CTO already added functionality to the download task to ensure that
all form attachments were downloaded before the form was copied into the
/sdcard/odk/forms directory and became 'live'. So this would be an extra
validity check before that wholesale move, allowing your non-upgraded
devices to stay 'in the past' w.r.t. form definitions they already have
until you upgrade the device.

This prevents catastrophic failure of an individual device, but does not
ensure complete migration of all the devices to the latest ODK Collect.

================
Device Fleet Management

ODK Aggregate, at least, does not have any mechanism to track all devices
and what their ODK Collect versions are. This is something that ODK Manage
was intended to do, but that never got implemented.

I haven't looked, but there are undoubtedly 3rd party tools that can
manage fleets of devices and communicate their configurations back to a
central location, and force-push updates onto individual devices. All the
anti-theft phone-lockout tools are basically doing that.

================
Form Revision Management

You do not need to upgrade all the forms at once.
Because the revision flags are in each form, you can keep running the
forms that require the bugs.
Even as you update your XLSForm, server and ODK Collect.

Depending upon how end-to-end your solution is, when running XLSForm to
update an existing form_id, you could access your live data collection
server to retrieve the existing form_id, and then, by default, apply the
same revision tags when making updates to that form_id. And only change
those revision tags through explicit user action.

That would allow users to easily edit the form to add another language or
fix a typo without much thought. It could also be expanded to run the
verification of identical data-models at that time (rather than later when
you try to upload the revision to the server) and to confirm that the form
version is appropriately larger than the existing form.

On Wed, Dec 2, 2015 at 4:37 PM, Christopher Robert crobert@surveycto.com wrote:

Thanks all, I appreciate all of the thinking going into this.

At the risk of thoroughly exhausting everybody's patience, I'm still not
sure that we have a strategy that can be described as reasonable from an
existing user's point of view. Let me explain why.

Say we release SurveyCTO 2.10 with three changes:

  1. We add the fix to JR and have it apply whenever the appropriate flag
    is set in the XForm model.

  2. We default all pyxform generation (which is basically invisible to the
    user in SurveyCTO) to include the appropriate flag so that newly-generated
    XML will be interpreted with the fixed JR.

  3. We default pyxform to output XPaths in choice-filter expressions using
    current() so that within-context repeat references finally work. This code,
    by default, relies on the fix (so paths are relative to the field, not the
    group). (It also changes ALL XPath expressions, not just those referencing
    within-repeat values, because we're not clever enough to be able to
    distinguish the different cases in that python code.)

Now say you're an upgrading user and you upgraded because SurveyCTO 2.10
also included a beautiful and astonishingly powerful web-based form
designer. You upgraded to get this new designer and you love it and now you
upload a new or revised form. That form happens to have a choice-filter
expression. Now here's where things get ugly...

That form won't work on your SurveyCTO 2.01, 2.00, 1.402, etc. devices.
Probably there's some nasty XPath parsing error when you get to where the
choice filter kicks in, maybe it even crashes your Collect. That form might
have worked the day before, and maybe all you changed was fixing a typo in
a label, but now suddenly it doesn't work on all of these devices.

So here's where we say, okay, the user has two choices: (1) upgrade all
devices to the 2.10 Collect, or (2) find and set the "stay in the past"
option. The second option is good in principle -- because it provides a
kind of quick save -- but then what does it take to ever migrate away from
that "stay in the past" setting? Not only do you need to then update 100%
of your devices with a high degree of certainty, but you also need to
re-generate the XML for every form on your server. And not even that is
enough. Then you need to update every form definition on every device.

And if you miss a few devices and leave them on the old version, or miss
a few form definitions here or there? Cryptic errors, inability to finalize
forms, confusion, anger, frantic calls to support, etc., etc.

You don't need to be a human-centered design expert to see that this is
kind of a disaster from a user-experience point of view. And, for us, this
would be multiplied across hundreds of migrating users, essentially eating
a lot of our 2016 and causing a tremendous amount of ill will.

The only upgrade path for existing users that makes any sense is one
where we basically emit the "stay in the past" flag always for everyone.
You can all turn your noses up at the idea of indefinite reliance on what
was initially a bug, and I get it. But at least here at SurveyCTO, we're
not going to put our users through the kind of migration path described
above.

If relying on the bug is just too unpalatable, then ultimately I don't
think that we can change pyxform to emit XPaths using current(). It's just
too fraught and we don't have a good way to manage the upgrade path without
torturing existing users. So fine, our awesome pyxform fix to magically
make within-repeat references start working is just not destined to see the
light of day. Rather, we need to make pyxform smart enough to only use
current() for paths to repeated values (which never worked in old JR's
anyway). Or we need to just provide a new solution for referencing repeated
values without current() (perhaps a new custom JR function, which everybody
loves).

And in that case, sure, I'm much more relaxed about the strategies for
upgrading users since very few are likely to be using current(). I won't
want to overly torture those users, but I won't lose a ton of sleep over
them since even users who use current() now are unlikely to use it in a lot
of forms.

I'll see if we can figure out how to make the pyxform change in such a
way that it only changes the XPaths for fields within repeat groups. But I
might also see if we can instead find a totally different way to solve or
work around this problem. Or, I'll see how I feel about pushing this off
for another year. Fixing choice filters to be able to work will within
repeat groups was not meant to be a huge project.

Thanks again,

Chris

On Wed, Dec 2, 2015 at 2:48 PM Mitch Sundt mitchellsundt@gmail.com wrote:

w.r.t. moving forward...

Even if this particular change impacts a very small number of forms, we
should figure out how to implement non-backward-compatible changes so that
they are safe moving forward. We have no visibility into who uses ODK
Collect and ODK Aggregate and no reliable way to notify them of changes.

This could be a huge deal when or if we ever revise the JR engine to
properly handle timezones, dates and times, and date and time arithmetic.

Or if we find a need to change XLSForm to emit XForms features or XPath
functions that currently don't work (e.g., to use [1] or position(), or
XForms events, for example).

Note that these non-backward-compatible changes extend to not just
XLSForm and ODK Collect, but also to servers. You can use the same revision
tag to indicate new or different processing to be taken on the server.

=============
Managing this change requires:

I believe that this requires:

(1) the XML form definition is the place to define the
feature-enabling-labels that the form requires of the device and server
processing steps.

(2) when the JR engine or any tool implements non-backward-compatible
changes, it only activates them when it detects the presence of those
feature-enabling-label in the XML form definition. Omitting that
feature-enabling-label is a directive to "stay in the past".

(3) ODK Collect and other tools processing the XML form definition or
the XML submission documents should warn the user (e.g., pop up a Toast) if
they are running a version that does not recognize the
feature-enabling-labels in the form's revision tag (i.e., they are given a
form that indicates an advanced feature that they are unaware of) -- this
notification informs the user that they should upgrade that tool to
correctly process the directive. Alternatively, the tool could reject the
form, Force Close or abort, which is the current behavior when ODK
Aggregate or ODK Collect gets an XForms definition that uses a new,
unrecognized, function, etc. ;-<

=============
The stable upgrade path is to:

(1) update your form distribution and data aggregation server(s),
(2) update your collection devices (or collection processing server --
e.g., Enketo), and
(3) only then update the form generation tool to emit forms with the
non-backward-compatible change.

=============
Specifying the Revision settings:

I believe the XLS file should not contain a setting to select or
specify the revision
.

I believe this should be specified at the time the XLSForm converter is
executed on the source XLS file.
I.e., the web UI for these tools should have a spinner or other UI
element that enables users to choose to "stay in the past"

This forces users to actively stay in the past.

The ODK site will probably achieve this with a drop-down that allows the
user to specify, e.g.,

"full current functionality"
"generate a form definition compatible with ODK Collect 1.4.7 and
earlier"

To allow all groups to independently advance the functionality within
their tool families, the combination of features that each of these
drop-down choices corresponds to should be documented by those groups.

I.e., I see a future where a form generated on Ona for Enketo may not be
compatible with stock ODK Collect unless an appropriate compatibility
drop-down is selected.

This could allow, for example, Survey CTO to implement and recoup costs
for JR date-time fixes or other XForms extensions within their ecosystem
before releasing that fix back to the wider community.

I do not see any value in offering a user the choice of which features
are enabled or disabled. I suppose some sites could offer that, but it
seems overly complex. Users really just need a way to specify compatibility
to whatever older version of the tools they are using.

==============
XLSForm functionality

I am fine with the XLSForm generator taking a comma-separated list of
features to enable, and passing that list through via the revision tag to
the the XML form definition.

Upon reflection, I think this is the only way to manage multiple
independent parallel contributions to the various codebases. It allows
different groups to add and enable different functionalities at different
rates.

We just need to maintain a global list of all of the feature labels that
every group is using so that we don't choose names that collide.

And, some of these feature labels may simply pass through the XLSForm
generator and on to the downstream tools. e.g., date-time fixes might not
have any impact on XLSForm, but would alter how the JR engine processed the
form.

On Wed, Dec 2, 2015 at 10:25 AM, Martijn van de Rijdt < martijn@enketo.org> wrote:

Ok, I was just wondering if it's really worth the effort. KoBo found 3
forms using current(). Not sure how many are actively used still. For them
it would seem to me easier to just contact that/those user(s) individually
to come up with a plan.

I can live with the XForm attribute and XLSForm setting addition.

Chris, to respond to your question about adding support for the bug to
Enketo. I don't currently see a reason to do so, as no Enketo user has ever
relied on the bug and I highly doubt any will in the future. However, I
don't think this should be relevant to this discussion. We're all in
agreement that we want the community to move forward with the correct
syntax.

On Tuesday, December 1, 2015 at 4:58:45 PM UTC-7, Christopher Robert wrote:

We have individual users with 500+ devices in the field. Updating
devices -- or even keeping track of which devices are on which versions --
is a major undertaking. In total, we're talking many thousands of devices
spread across many countries. The problem is not a small one, and our
commitment to backward compatibility is one that our users not only
appreciate but expect.

ODK/SurveyCTO Collect continues to use older versions of forms to
view, edit, or finalize submissions begun under those older versions, even
when newer versions exist. So the backward-compatibility challenge is a
formidable one. If you change the way JavaRosa interprets expressions in
the kind of blanket way that you suggest, then users will be faced with the
impossible choice of running software that fails to function properly with
existing, in-progress, or old submissions, or running software that fails
to function properly with newer submissions. Multiply the difficulty of the
choice and the potential confusion of XPath parsing errors times
however-many devices there are in the field.

No, this is a legitimately difficult situation that requires a
tremendous amount of care. Changing how XPath expressions are interpreted
is a big deal for the Collect world, though I can see why it would be
considerably less so in the Enketo world.

Now, I will admit that there are bound to be very few users who
already use current() in choice filters. Mitch suggested the approach a few
times in the public forum, but even I failed to get it to work the first
several times I tried to follow his advice. So chances are that not many
projects are using current() today. (I know of 1-2, maybe there are 8-10
total. That's just a guess.)

Enketo aside, we could -- today! -- allow people to reference
in-context repeated values by exploiting the current JR support for
current(). For some users/projects, there would be an immediate benefit.
But obviously that relies on "the bug" and so it would expand the potential
need for backward compatibility. And it would break choice-filter
compatibility between Collect and Enketo (say, for SurveyCTO users at
least).

It's easy to say "just fix the bug and then this problem goes away,"
but the fact is that we don't quite know how to fix it without breaking
other things; we have a candidate for a fix, but it's seriously scary. It
may take time. And meanwhile, we have already gone multiple years with
users waiting to be able to reference within-context repeated values in
choice filters. If a safe fix drags on for another year or two -- either
because it's just too hard or because nobody quite has the time -- is that
just the way it goes? I don't know. It's not obvious to me that breaking
compatibility with Enketo and relying on current behavior isn't the
welfare-maximizing choice.

In the end, I believe that Mitch's suggestion re: compatibility is
maybe necessary (storing in the XML some indication of how current() in
choice filters should be interpreted). And if we totally screwed up the fix
and broke a bunch of use cases, at least it might offer some escape for
affected users (if we allowed an override on the settings sheet).

So no, I don't agree that it's as simple as "fix the bug and make
people upgrade."

Chris

On Tue, Dec 1, 2015 at 5:11 PM Martijn van de Rijdt mar...@enketo.org wrote:

I can't help thinking that we're overthinking this
backwards-compatibility issue.

Ideally we have old clients that have the bug and new clients that do
not have the bug. Isn't that the case with any other major bug fix? If an
old form stops working in the new version of ODK Collect, the survey owner
will either have to upload a new version of the form or the user will have
to downgrade ODK Collect. Surely this will be a problem for some users, but
for how many? A small fraction of one percent seems like a reasonable guess.

Maybe the backwards-compatibility should be addressed just by warning
users, as Yaw suggested way above.

Any 'side-effects' of fixing the bug, like with creating itemsets,
can be tested for, I presume, before deploying the fix.

One way to detect at least a significant portion of the current
current() users, is to do a regex search for any usage of the
'current()/path/to/node' where current()/ is not followed by '../'. That
usage is specific to this bug to refer to a sibling node and will not occur
any more after the bug is fixed (because a question cannot have children).

On Mon, Nov 30, 2015 at 11:43 AM, Christopher Robert < cro...@surveycto.com> wrote:

I wasn't going to suggest adding the different parameters to any of
the XLSForm templates -- so they wouldn't become cluttered with this stuff.
Really, it would just be an emergency option for somebody requiring
different compatibility for some component that was out of sync with their
other components. Ideally, nobody would actually have to use the settings
sheet to override any of these options.

To be specific, my proposal would be that everybody
support current_always_field=false today, and that this be how pyxform
today defaults. Ideally, this would include Enketo, as Collect and
SurveyCTO's Web Collect (and anything else using the ODK JavaRosa) already
support current_always_field=false as their current behavior. Then, we
figure out how to get current_always_field=true to work in JavaRosa and
phase that in, eventually changing the pyxform default to that once we
think that enough components support it.

We have thousands of devices out there using older Collect versions,
even well after their servers have been updated... so my guess is that it's
some of those users who will end up having to use the settings-sheet option
to force current_always_field=false after we've changed the default to true.

Does all of that make sense?

Chris

On Sun, Nov 29, 2015 at 6:48 PM Yaw Anokwa yan...@nafundi.com wrote:

All this sounds fine to me, but wanted to raise some potential
issues.
Discrete parameters also means folks can pick and choose what
behavior
they want and it's hard to move the entire community forward that
way.
Also, once you get past two or three, you end up with a settings
sheet
of lots of flags to toggle and that isn't a great user experience.

Yaw

Need ODK consultants? Nafundi provides form design, server setup,
in-field training, and software development for ODK. Go to
https://nafundi.com to get started.

On Sun, Nov 29, 2015 at 6:47 PM, Christopher Robert cro...@surveycto.com wrote:

Having a number of discrete parameters that are overridable on
the settings
sheet makes sense to me (like "current_references_parent" or
perhaps
"current_always_field"). We could then add model parameters and
change the
pyxform default over time, but users could always override on
their settings
sheets as needed for specific compatibility.

For example, our SurveyCTO pyxform for 2.10 (we've given up on
2.01 now)
could default to current_always_field=false, which would mean
that JR would
be backward-compatible and pyxform would emit relative paths for
the
choice_filter column assuming that current() references the
parent. This
would break all choice_filters in Enketo, but our Enketo users
could set
current_always_field=true in their settings sheets.

For that to work, though, we'd also have to add support for
current_always_field=true in JR. We could do that as Meletis has
proposed,
which may have other side effects -- we'll have to see (e.g., for
itemset
copying).

Alternatively, we could try to convince Martijn to support
current_always_field=false, which is likely trivial. If he would
do that,
then we could push JR's support for current_always_field=true to
when we
could figure out how to do it safely. Also, it would mean
producing a
version of Enketo that supports relative paths in choice filters
in a manner
compatible with ODK Collect. Ona, Kobo, and other Enketo users
could
rejoice, able to produce forms that use current() (like those
that reference
same-repeat values) in their choice_filters. That boon to users
has to be
worth something, even if a principle is slightly violated
(supporting
something outside the XPath spec).

Chris

On Tue, Nov 24, 2015 at 8:43 PM Alex Dorey dore...@gmail.com wrote:

Hi, Just catching up on the discussion. Sorry if I'm a bit
behind in my
understanding of things.

Regarding the pyxform changes--

  1. Instead of the date, could we have the configuration
    parameter describe
    the specific change?

So instead of:

revision="2015-11-24"

we specify something like:

current_references_parent=True (default to False with the next
pyxform
version bump)

  1. Could we put this configuration into the xlsform's settings
    sheet? I
    think this would be easier to test and configure. We could also
    add a way to
    override the setting with an extra argument to the CLI or
    xls2json call.

On Tuesday, November 24, 2015 at 3:26:18 PM UTC-5, Mitch wrote:

I agree. Having just delved into the itemset code, there are a
number of
issues that would need to be cleaned up.

What I would propose is to leave JR / ODK Collect broken
as-is. This
will be future work.

The python code will then be the focus of the tweaks.

(1) change:

json_survey =

xls2json.workbook_to_json(json.loads(request.POST['workbookJson']),

form_name=form_name, warnings=warningsList)

to add an optional revision argument:

json_survey =

xls2json.workbook_to_json(json.loads(request.POST['workbookJson']),

form_name=form_name, warnings=warningsList,
revision='2015-11-24')

(2) change that implementation to:

(2a) store the revision string somewhere in the returned json
object.

(2b) expand ${fieldname} with the new relative path expansion
only if the
passed-in revision is >= '2015-11-24' ; otherwise, use the
absolute path
expansion.

(3) change:

create_survey_element_from_dict()

to emit the revision value as an attribute on the
element.

===============
The JR itemset bug will be fixed at a later date.

When that is fixed, we can introduce a new revision string for
that
behavior change.

i.e., once we have the revision string moving through the
system, we can
begin to support non-backward-compatible changes to behaviors
in XLSForm and
JR form processing.

For the time being, the ODK XLSForm converter will not supply a
revision
argument, and not make use of this pyxform enhancement. Once
the JR fix is
done, we will then support that new revision and the current
behavior,
defaulting to the new behavior, but supporting users in
generating the old
behavior.

Mitch

On Tue, Nov 24, 2015 at 8:44 AM, Martijn van de Rijdt < mar...@enketo.org> wrote:

I appreciate the creative solution. It does sound like it will
become a
bit of a headache to manage and explain to users though.
Whether it's worth
it depends on how many active forms are out there that rely on
the bug. We
should get Ona's and KoBo's input on this too, I think.

From Enketo's side, relative paths were never supported in
choice-filters (to my embarrassment and surprise - which led
to my
question). As a result, there is no significant issue with
backwards
compatibility in Enketo once it is fixed in JR. Not sure how
meaningful this
is to ODK Collect, but no Enketo user has ever reported this
lack of
support. I think to date it's been too hard to craft such
choice-filters.
Because of the way Enketo is integrated into KoBo, Ona,
Formhub, I'd be
surprised if there were a significant amount of those users
using relative
choice-filters.

I do indeed feel strongly about following the XForms 2.0 spec
here. I
think relative path support is going to be very useful in the
future, and I
agree that we need to look at the bigger picture and do this
correctly, so
it won't catch up with us.

On Tue, Nov 24, 2015 at 4:24 AM, Christopher Robert cro...@surveycto.com wrote:

Okay, well, this is turning into a bigger and bigger project
and we may
well miss the window for our 2.01 maintenance release (and/or
make the
changes so extensive and scary that we cannot in good
conscience include
them in a maintenance release).

However, your idea for making JR sensitive to the revision
value
emitted by pyxform opens a new possibility: we could release
the pyxform
change with 2.01 so that users can begin to benefit from the
relative paths
(relying on today's current() behavior), then assume that
when JR is "fixed"
it will only be fixed for XML emitted by newer pyxforms that
would have
updated their paths to account for the fix (using your
revision-value idea).
The primary drawback to this approach is that we would
basically break all
cascading-select filter expressions for SurveyCTO+Enketo
users (since
Martijn clearly wants to implement the "fixed" version of
current() rather
than the JavaRosa-compatible version of current()).

Meletis also found another issue relating to the evaluation
context in
jr:choice-name() (
https://github.com/opendatakit/opendatakit/issues/1180),
so we have that to contend with as well. If we can't safely
resolve the
context for jr:choice-name(), then that would present another
major drawback
and further lessen our enthusiasm for this entire adventure.

Best,

Chris

On Mon, Nov 23, 2015 at 5:45 PM Mitch Sundt < mitche...@gmail.com> wrote:

And, if it isn't clear, to preserve backward-compatible
operations,
XLSForm should not expand ${fieldname} with a relative path
unless the
revision value is the newer one; if the revision is not
specified, it would
emit absolute paths like it currently does.

I also switched to use a date value for the revision value
("2015-11-23") which allows simple lexical comparisons.

On Mon, Nov 23, 2015 at 1:03 PM, Mitch Sundt < mitche...@gmail.com> wrote:

w.r.t. backward compatibility

I too am concerned.

A proposed solution:

  1. Add a command line argument or other mechanism to
    request that
    XLSForm inject a 'revision="2015-11-23"' attribute into the
    element.
    e.g.,

This is used to alter the behavior of the XForms engine to
accommodate backward-compatible bug behaviors.

We can't identify the bugs (e.g,
'legacy="itemset-parent"'), since
that would break older forms run on the new ODK Collect
releases.
We have to advance the revision code as bugs are fixed.

I think an integer value is fine for this.

I don't anticipate making a lot of revisions, and I think
the
revisions would be sequentially applied (you can't
pick-and-choose what bugs
you include).

In the future, this would allow us to re-do the date-time
behavior.

  1. Modify the XLSForm tools (e.g., the online form
    generator) to
    enable the user to select what revision they want to emit
    (much like we did
    last year with the JR form eval choice setting within ODK
    Collect). We can
    default to emitting the latest revision if the user does
    not make a choice.

  2. Augment JR to look for this revision attribute and
    store/restore
    it in the FormDef binary object.

Based upon this value, alter the behavior of the JR itemset
evaluation to fix the path bug.

  1. Modify ODK Validate to scan for syntax that may be
    impacted by a
    revision (to advise whether to change something). Not sure
    how complex that
    might be. May not be always possible?

w.r.t. impacts of change

Thank you Meletis for researching when this change occurred.

I will have to ponder what effect the parent/node change
would have
on itemset copying. Hadn't thought that there was any
relationship between
the two, but perhaps there is.

Yaw also reported that the Cascading Select itemsets on
opendatakit.appspot.com are not working with the 1.4.7 rev
1053 release (and
they work but give odd 'null' resolutions on 1057). This is
in a related
area of this same code, so I will hopefully gain a better
understanding of
dynamic itemsets as I uncover the causes of that problem.


Mitch

On Mon, Nov 23, 2015 at 5:59 AM, Christopher Robert cro...@surveycto.com wrote:

So just to be clear, the "fix" we're testing here reverts
the entire
context within that populateDynamicChoices() function, to
be the field
instead of the parent -- so the effects are potentially
much wider than just
what current() returns. Potentially anything and
everything that depends on
context can change now, since the context will shift from
parent to field.

This means that we revert a change from 5 years ago and
alter the
context from now on. Neither Meletis nor I feel terribly
confident that we
can entirely predict the consequences to backward
compatibility. We're
testing the things we can think to test, but honestly it's
XML users in the
broader ODK community who face the greatest risks here;
SurveyCTO users by
and large use XLSForm and thus the potential XPath syntax
consequences are
sharply limited.

Are you guys very sure that you want to take this path? I
can tell
you now that if we make this change and then there is some
outcry after the
next ODK Collect release, SurveyCTO Collect will very
likely stick with the
new change (since we will have changed pyxform and we'll
then have too many
backward-compatibility concerns to then reverse the
change).

Thanks,

Chris

On Mon, Nov 23, 2015 at 5:23 AM mel...@surveycto.com wrote:

Just for the history's sake, I found the time where the
behavior
changed. It was at March 11, 2010:

https://bitbucket.org/m.sundt/javarosa/commits/b41875b2c23c9a21a868a5d2dc0ffc1f6b5b0b65#Lcore/src/org/javarosa/core/model/FormDef.javaT635

Take a look only in the changes of the
"populateDynamicChoices"
method. The context reference used to be the "current"
field, but after the
changes above it started be the field's parent (by adding
getParent()).
Perhaps there was a reason for that change, but perhaps
not, but I think all
of us agree that this should be fixed, even after 5 and a
half years.

Meletis

On Monday, November 23, 2015 at 12:08:47 PM UTC+2, mel...@surveycto.com wrote:

Hi Mitch,

Please take a look at the code comment in line 31 here:

https://bitbucket.org/m.sundt/javarosa/src/bf17fb749b74e53915f086ca3827c3c6fe73fd07/core/src/org/javarosa/core/model/ItemsetBinding.java?at=default&fileviewer=file-view-default#ItemsetBinding.java-31

It clearly says: "ref of the control parent
(group/formdef) of
itemset question"

So, for some reason, they wanted it to point to the
parent. I also
see that when the dynamic choices are populated, the
evaluation context is
wrapped around that parent (contextRef object), that's
why current() refers
to the parent group when evaluating itemsets nodesets.

I will run a few tests by changing the contextRef field
value, I
just wanted to mention that in case you remember a
reason for that code
comment.

Meletis

On Friday, November 20, 2015 at 8:49:12 PM UTC+2, Mitch wrote:

Yes, I think it should be a one-line change; just don't
know
where.

Mitch

(I do jar-updates over Christmas / New-Years. Tedious,
but
doesn't require a whole lot of attention.)

On Fri, Nov 20, 2015 at 10:36 AM, mel...@surveycto.com wrote:

Hi Mitch,

Thank you for all the details, I guess I was missing
some of the
context of the original issue.

We will give it a shot to see if we can come up with a
few ideas
how to fix the evaluation content of the itemset
nodeset. I just hope it
will not be like last year with the "form processing
logic" in JR, otherwise
I believe that every Christmas will be a JR Christmas.
:slight_smile:

Meletis

On Friday, November 20, 2015 at 7:29:14 PM UTC+2, Mitch wrote:

The form was attached in one of my earlier replies.

On Fri, Nov 20, 2015 at 9:27 AM, Mitch Sundt mitche...@gmail.com wrote:

And you see this in constraint expressions where you
write:

. > 6

I.e., current()/. should refer to the field, and not
the group
containing the field, as it currently does when
evaluating expressions in
the itemset nodeset.

On Fri, Nov 20, 2015 at 9:26 AM, Mitch Sundt mitche...@gmail.com wrote:

The issue is that there is a discrepancy in the
meaning of
'.' across the different contexts of: calculate,
relevant, , and
these itemset nodeset expressions.

In other usages, you need to reference this other
field with
current()/../filterfield ; but in the nodeset
expression, you would
reference it with current()/./filterfield

I.e., the notion of '.' is different when used
within a
nodeset expression and when used within a
constraint expression or an
field.

See the attached NBiggestOfSet form. For the
label, it has:

    <select1

ref="/NBiggestOfSet/education/edu_level">

      <label>What is <output

value="current()/../referred_attendee_name"/>'s
level of education?
...

For a constraint, it has:
<bind

calculate="/NBiggestOfSet/attendee/name[position(..)=int(current()/../referred_attendee)]"

nodeset="/NBiggestOfSet/education/referred_attendee_name" readonly="true()"

type="string"/>

These each use the current()/../ to reference the
group (
/NBiggesetOfSet/education ).

But, within the nodeset definition (if this form
used one),
you would need to use
current()/./referred_attendee_name to reference a
field in that group.

On Fri, Nov 20, 2015 at 8:22 AM, < mel...@surveycto.com> wrote:

Hi Mitch,

I think I'm missing something but I think that
nothing is
broken and that the only issue is that the
expression
current()/../filterfield in your example above
should be just changed to
current()/filterfield. If you make that change,
everything should work
because it will be resolved to:

/data/group1/filterfield

Why should a "../" be present there? We have used
current()/somefield successfully in one of our
test forms and it works fine.
And I think it would also work great in your
example. Wouldn't it? Or am I
totally out of context?

Meletis

On Friday, November 20, 2015 at 5:35:16 PM UTC+2, Mitch wrote:

Hi Meletis,

That's not the issue --

The issue is that if you use a filter condition
in the
&g

--
Revolutionizing data collection since 2012.

Enketo https://enketo.org/ | LinkedIn
http://www.linkedin.com/company/enketo-llc | GitHub
https://github.com/enketo | Twitter
https://twitter.com/enketo | Blog http://blog.enketo.org/

--
You received this message because you are subscribed to the Google
Groups "ODK Developers" group.
To unsubscribe from this group and stop receiving emails from it, send
an email to opendatakit-developers+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--
Mitch Sundt
Software Engineer
University of Washington
mitchellsundt@gmail.com

--
You received this message because you are subscribed to the Google
Groups "ODK Developers" group.
To unsubscribe from this group and stop receiving emails from it, send
an email to opendatakit-developers+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups
"ODK Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to opendatakit-developers+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--
Mitch Sundt
Software Engineer
University of Washington
mitchellsundt@gmail.com

--
You received this message because you are subscribed to the Google Groups
"ODK Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to opendatakit-developers+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

FYI, I've just found a way to have pyxform only use current() when
replacing ${x} references that (a) are within choice_filter expressions and
(b) point to groups or fields within repeat groups. We're going to do a
bunch of testing to see how robust that change is (in terms of not breaking
any other cases and in terms of not imposing any performance costs). If it
looks solid, then it will become a candidate for inclusion in our 2.01
release.

That limits possible damage to cases that were already broken, and shifts
them to function based on the legacy JR support for current(). Our few
Enketo users won't benefit from the new ability to reference repeated
values in choice_filter expressions, but I think that's okay because
they're not losing anything. And any future fix to JR's current() behavior
will just need to be backward-compatible as discussed, based on flags in
the form model.

We'll be happy to share the pyxform changes with the community, if the
broader community would like to have this option to emit functional
choice_filter expressions for repeated fields. And the pyxform is easily
changed to emit XPaths compatible with the future-desired current()
behavior (though obviously the difficulty is in the backward compatibility,
as discussed at length).

Best,

Chris

··· On Fri, Dec 4, 2015 at 12:20 AM Christopher Robert wrote:

Thanks, Mitch. Some great ideas in there.

Best,

Chris

On Thu, Dec 3, 2015 at 12:52 PM Mitch Sundt mitchellsundt@gmail.com wrote:

It isn't that dire.

Not only do you need to then update 100% of your devices with a high
degree of certainty, but you also need to re-generate the XML for every
form on your server. And not even that is enough. Then you need to update
every form definition on every device.

=================
Individual Device Safeguards

Yes, all the devices need to be updated. And yes, this is the biggest
pain point. We CAN change the form download task so that it first scans
the form to detect unsupported revision tags, and refuses to install the
form definition on the device. And then throws up a Dialog warning the user
of the inappropriate form and prodding them to update ODK Collect. This
would preserve the older version of that individual form on that one
device. Survey CTO already added functionality to the download task to
ensure that all form attachments were downloaded before the form was copied
into the /sdcard/odk/forms directory and became 'live'. So this would be an
extra validity check before that wholesale move, allowing your non-upgraded
devices to stay 'in the past' w.r.t. form definitions they already have
until you upgrade the device.

This prevents catastrophic failure of an individual device, but does not
ensure complete migration of all the devices to the latest ODK Collect.

================
Device Fleet Management

ODK Aggregate, at least, does not have any mechanism to track all devices
and what their ODK Collect versions are. This is something that ODK Manage
was intended to do, but that never got implemented.

I haven't looked, but there are undoubtedly 3rd party tools that can
manage fleets of devices and communicate their configurations back to a
central location, and force-push updates onto individual devices. All the
anti-theft phone-lockout tools are basically doing that.

================
Form Revision Management

You do not need to upgrade all the forms at once.
Because the revision flags are in each form, you can keep running the
forms that require the bugs.
Even as you update your XLSForm, server and ODK Collect.

Depending upon how end-to-end your solution is, when running XLSForm to
update an existing form_id, you could access your live data collection
server to retrieve the existing form_id, and then, by default, apply the
same revision tags when making updates to that form_id. And only change
those revision tags through explicit user action.

That would allow users to easily edit the form to add another language or
fix a typo without much thought. It could also be expanded to run the
verification of identical data-models at that time (rather than later when
you try to upload the revision to the server) and to confirm that the form
version is appropriately larger than the existing form.

On Wed, Dec 2, 2015 at 4:37 PM, Christopher Robert <crobert@surveycto.com wrote:

Thanks all, I appreciate all of the thinking going into this.

At the risk of thoroughly exhausting everybody's patience, I'm still not
sure that we have a strategy that can be described as reasonable from an
existing user's point of view. Let me explain why.

Say we release SurveyCTO 2.10 with three changes:

  1. We add the fix to JR and have it apply whenever the appropriate flag
    is set in the XForm model.

  2. We default all pyxform generation (which is basically invisible to
    the user in SurveyCTO) to include the appropriate flag so that
    newly-generated XML will be interpreted with the fixed JR.

  3. We default pyxform to output XPaths in choice-filter expressions
    using current() so that within-context repeat references finally work. This
    code, by default, relies on the fix (so paths are relative to the field,
    not the group). (It also changes ALL XPath expressions, not just those
    referencing within-repeat values, because we're not clever enough to be
    able to distinguish the different cases in that python code.)

Now say you're an upgrading user and you upgraded because SurveyCTO 2.10
also included a beautiful and astonishingly powerful web-based form
designer. You upgraded to get this new designer and you love it and now you
upload a new or revised form. That form happens to have a choice-filter
expression. Now here's where things get ugly...

That form won't work on your SurveyCTO 2.01, 2.00, 1.402, etc. devices.
Probably there's some nasty XPath parsing error when you get to where the
choice filter kicks in, maybe it even crashes your Collect. That form might
have worked the day before, and maybe all you changed was fixing a typo in
a label, but now suddenly it doesn't work on all of these devices.

So here's where we say, okay, the user has two choices: (1) upgrade all
devices to the 2.10 Collect, or (2) find and set the "stay in the past"
option. The second option is good in principle -- because it provides a
kind of quick save -- but then what does it take to ever migrate away from
that "stay in the past" setting? Not only do you need to then update 100%
of your devices with a high degree of certainty, but you also need to
re-generate the XML for every form on your server. And not even that is
enough. Then you need to update every form definition on every device.

And if you miss a few devices and leave them on the old version, or miss
a few form definitions here or there? Cryptic errors, inability to finalize
forms, confusion, anger, frantic calls to support, etc., etc.

You don't need to be a human-centered design expert to see that this is
kind of a disaster from a user-experience point of view. And, for us, this
would be multiplied across hundreds of migrating users, essentially eating
a lot of our 2016 and causing a tremendous amount of ill will.

The only upgrade path for existing users that makes any sense is one
where we basically emit the "stay in the past" flag always for everyone.
You can all turn your noses up at the idea of indefinite reliance on what
was initially a bug, and I get it. But at least here at SurveyCTO, we're
not going to put our users through the kind of migration path described
above.

If relying on the bug is just too unpalatable, then ultimately I don't
think that we can change pyxform to emit XPaths using current(). It's just
too fraught and we don't have a good way to manage the upgrade path without
torturing existing users. So fine, our awesome pyxform fix to magically
make within-repeat references start working is just not destined to see the
light of day. Rather, we need to make pyxform smart enough to only use
current() for paths to repeated values (which never worked in old JR's
anyway). Or we need to just provide a new solution for referencing repeated
values without current() (perhaps a new custom JR function, which everybody
loves).

And in that case, sure, I'm much more relaxed about the strategies for
upgrading users since very few are likely to be using current(). I won't
want to overly torture those users, but I won't lose a ton of sleep over
them since even users who use current() now are unlikely to use it in a lot
of forms.

I'll see if we can figure out how to make the pyxform change in such a
way that it only changes the XPaths for fields within repeat groups. But I
might also see if we can instead find a totally different way to solve or
work around this problem. Or, I'll see how I feel about pushing this off
for another year. Fixing choice filters to be able to work will within
repeat groups was not meant to be a huge project.

Thanks again,

Chris

On Wed, Dec 2, 2015 at 2:48 PM Mitch Sundt mitchellsundt@gmail.com wrote:

w.r.t. moving forward...

Even if this particular change impacts a very small number of forms, we
should figure out how to implement non-backward-compatible changes so that
they are safe moving forward. We have no visibility into who uses ODK
Collect and ODK Aggregate and no reliable way to notify them of changes.

This could be a huge deal when or if we ever revise the JR engine to
properly handle timezones, dates and times, and date and time arithmetic.

Or if we find a need to change XLSForm to emit XForms features or XPath
functions that currently don't work (e.g., to use [1] or position(), or
XForms events, for example).

Note that these non-backward-compatible changes extend to not just
XLSForm and ODK Collect, but also to servers. You can use the same revision
tag to indicate new or different processing to be taken on the server.

=============
Managing this change requires:

I believe that this requires:

(1) the XML form definition is the place to define the
feature-enabling-labels that the form requires of the device and server
processing steps.

(2) when the JR engine or any tool implements non-backward-compatible
changes, it only activates them when it detects the presence of those
feature-enabling-label in the XML form definition. Omitting that
feature-enabling-label is a directive to "stay in the past".

(3) ODK Collect and other tools processing the XML form definition or
the XML submission documents should warn the user (e.g., pop up a Toast) if
they are running a version that does not recognize the
feature-enabling-labels in the form's revision tag (i.e., they are given a
form that indicates an advanced feature that they are unaware of) -- this
notification informs the user that they should upgrade that tool to
correctly process the directive. Alternatively, the tool could reject the
form, Force Close or abort, which is the current behavior when ODK
Aggregate or ODK Collect gets an XForms definition that uses a new,
unrecognized, function, etc. ;-<

=============
The stable upgrade path is to:

(1) update your form distribution and data aggregation server(s),
(2) update your collection devices (or collection processing server --
e.g., Enketo), and
(3) only then update the form generation tool to emit forms with the
non-backward-compatible change.

=============
Specifying the Revision settings:

I believe the XLS file should not contain a setting to select or
specify the revision
.

I believe this should be specified at the time the XLSForm converter is
executed on the source XLS file.
I.e., the web UI for these tools should have a spinner or other UI
element that enables users to choose to "stay in the past"

This forces users to actively stay in the past.

The ODK site will probably achieve this with a drop-down that allows
the user to specify, e.g.,

"full current functionality"
"generate a form definition compatible with ODK Collect 1.4.7 and
earlier"

To allow all groups to independently advance the functionality within
their tool families, the combination of features that each of these
drop-down choices corresponds to should be documented by those groups.

I.e., I see a future where a form generated on Ona for Enketo may not
be compatible with stock ODK Collect unless an appropriate compatibility
drop-down is selected.

This could allow, for example, Survey CTO to implement and recoup costs
for JR date-time fixes or other XForms extensions within their ecosystem
before releasing that fix back to the wider community.

I do not see any value in offering a user the choice of which features
are enabled or disabled. I suppose some sites could offer that, but it
seems overly complex. Users really just need a way to specify compatibility
to whatever older version of the tools they are using.

==============
XLSForm functionality

I am fine with the XLSForm generator taking a comma-separated list of
features to enable, and passing that list through via the revision tag to
the the XML form definition.

Upon reflection, I think this is the only way to manage multiple
independent parallel contributions to the various codebases. It allows
different groups to add and enable different functionalities at different
rates.

We just need to maintain a global list of all of the feature labels
that every group is using so that we don't choose names that collide.

And, some of these feature labels may simply pass through the XLSForm
generator and on to the downstream tools. e.g., date-time fixes might not
have any impact on XLSForm, but would alter how the JR engine processed the
form.

On Wed, Dec 2, 2015 at 10:25 AM, Martijn van de Rijdt < martijn@enketo.org> wrote:

Ok, I was just wondering if it's really worth the effort. KoBo found 3
forms using current(). Not sure how many are actively used still. For them
it would seem to me easier to just contact that/those user(s) individually
to come up with a plan.

I can live with the XForm attribute and XLSForm setting addition.

Chris, to respond to your question about adding support for the bug to
Enketo. I don't currently see a reason to do so, as no Enketo user has ever
relied on the bug and I highly doubt any will in the future. However, I
don't think this should be relevant to this discussion. We're all in
agreement that we want the community to move forward with the correct
syntax.

On Tuesday, December 1, 2015 at 4:58:45 PM UTC-7, Christopher Robert wrote:

We have individual users with 500+ devices in the field. Updating
devices -- or even keeping track of which devices are on which versions --
is a major undertaking. In total, we're talking many thousands of devices
spread across many countries. The problem is not a small one, and our
commitment to backward compatibility is one that our users not only
appreciate but expect.

ODK/SurveyCTO Collect continues to use older versions of forms to
view, edit, or finalize submissions begun under those older versions, even
when newer versions exist. So the backward-compatibility challenge is a
formidable one. If you change the way JavaRosa interprets expressions in
the kind of blanket way that you suggest, then users will be faced with the
impossible choice of running software that fails to function properly with
existing, in-progress, or old submissions, or running software that fails
to function properly with newer submissions. Multiply the difficulty of the
choice and the potential confusion of XPath parsing errors times
however-many devices there are in the field.

No, this is a legitimately difficult situation that requires a
tremendous amount of care. Changing how XPath expressions are interpreted
is a big deal for the Collect world, though I can see why it would be
considerably less so in the Enketo world.

Now, I will admit that there are bound to be very few users who
already use current() in choice filters. Mitch suggested the approach a few
times in the public forum, but even I failed to get it to work the first
several times I tried to follow his advice. So chances are that not many
projects are using current() today. (I know of 1-2, maybe there are 8-10
total. That's just a guess.)

Enketo aside, we could -- today! -- allow people to reference
in-context repeated values by exploiting the current JR support for
current(). For some users/projects, there would be an immediate benefit.
But obviously that relies on "the bug" and so it would expand the potential
need for backward compatibility. And it would break choice-filter
compatibility between Collect and Enketo (say, for SurveyCTO users at
least).

It's easy to say "just fix the bug and then this problem goes away,"
but the fact is that we don't quite know how to fix it without breaking
other things; we have a candidate for a fix, but it's seriously scary. It
may take time. And meanwhile, we have already gone multiple years with
users waiting to be able to reference within-context repeated values in
choice filters. If a safe fix drags on for another year or two -- either
because it's just too hard or because nobody quite has the time -- is that
just the way it goes? I don't know. It's not obvious to me that breaking
compatibility with Enketo and relying on current behavior isn't the
welfare-maximizing choice.

In the end, I believe that Mitch's suggestion re: compatibility is
maybe necessary (storing in the XML some indication of how current() in
choice filters should be interpreted). And if we totally screwed up the fix
and broke a bunch of use cases, at least it might offer some escape for
affected users (if we allowed an override on the settings sheet).

So no, I don't agree that it's as simple as "fix the bug and make
people upgrade."

Chris

On Tue, Dec 1, 2015 at 5:11 PM Martijn van de Rijdt < mar...@enketo.org> wrote:

I can't help thinking that we're overthinking this
backwards-compatibility issue.

Ideally we have old clients that have the bug and new clients that
do not have the bug. Isn't that the case with any other major bug fix? If
an old form stops working in the new version of ODK Collect, the survey
owner will either have to upload a new version of the form or the user will
have to downgrade ODK Collect. Surely this will be a problem for some
users, but for how many? A small fraction of one percent seems like a
reasonable guess.

Maybe the backwards-compatibility should be addressed just by
warning users, as Yaw suggested way above.

Any 'side-effects' of fixing the bug, like with creating itemsets,
can be tested for, I presume, before deploying the fix.

One way to detect at least a significant portion of the current
current() users, is to do a regex search for any usage of the
'current()/path/to/node' where current()/ is not followed by '../'. That
usage is specific to this bug to refer to a sibling node and will not occur
any more after the bug is fixed (because a question cannot have children).

On Mon, Nov 30, 2015 at 11:43 AM, Christopher Robert < cro...@surveycto.com> wrote:

I wasn't going to suggest adding the different parameters to any of
the XLSForm templates -- so they wouldn't become cluttered with this stuff.
Really, it would just be an emergency option for somebody requiring
different compatibility for some component that was out of sync with their
other components. Ideally, nobody would actually have to use the settings
sheet to override any of these options.

To be specific, my proposal would be that everybody
support current_always_field=false today, and that this be how pyxform
today defaults. Ideally, this would include Enketo, as Collect and
SurveyCTO's Web Collect (and anything else using the ODK JavaRosa) already
support current_always_field=false as their current behavior. Then, we
figure out how to get current_always_field=true to work in JavaRosa and
phase that in, eventually changing the pyxform default to that once we
think that enough components support it.

We have thousands of devices out there using older Collect
versions, even well after their servers have been updated... so my guess is
that it's some of those users who will end up having to use the
settings-sheet option to force current_always_field=false after we've
changed the default to true.

Does all of that make sense?

Chris

On Sun, Nov 29, 2015 at 6:48 PM Yaw Anokwa yan...@nafundi.com wrote:

All this sounds fine to me, but wanted to raise some potential
issues.
Discrete parameters also means folks can pick and choose what
behavior
they want and it's hard to move the entire community forward that
way.
Also, once you get past two or three, you end up with a settings
sheet
of lots of flags to toggle and that isn't a great user experience.

Yaw

Need ODK consultants? Nafundi provides form design, server setup,
in-field training, and software development for ODK. Go to
https://nafundi.com to get started.

On Sun, Nov 29, 2015 at 6:47 PM, Christopher Robert cro...@surveycto.com wrote:

Having a number of discrete parameters that are overridable on
the settings
sheet makes sense to me (like "current_references_parent" or
perhaps
"current_always_field"). We could then add model parameters and
change the
pyxform default over time, but users could always override on
their settings
sheets as needed for specific compatibility.

For example, our SurveyCTO pyxform for 2.10 (we've given up on
2.01 now)
could default to current_always_field=false, which would mean
that JR would
be backward-compatible and pyxform would emit relative paths for
the
choice_filter column assuming that current() references the
parent. This
would break all choice_filters in Enketo, but our Enketo users
could set
current_always_field=true in their settings sheets.

For that to work, though, we'd also have to add support for
current_always_field=true in JR. We could do that as Meletis has
proposed,
which may have other side effects -- we'll have to see (e.g.,
for itemset
copying).

Alternatively, we could try to convince Martijn to support
current_always_field=false, which is likely trivial. If he would
do that,
then we could push JR's support for current_always_field=true to
when we
could figure out how to do it safely. Also, it would mean
producing a
version of Enketo that supports relative paths in choice filters
in a manner
compatible with ODK Collect. Ona, Kobo, and other Enketo users
could
rejoice, able to produce forms that use current() (like those
that reference
same-repeat values) in their choice_filters. That boon to users
has to be
worth something, even if a principle is slightly violated
(supporting
something outside the XPath spec).

Chris

On Tue, Nov 24, 2015 at 8:43 PM Alex Dorey dore...@gmail.com wrote:

Hi, Just catching up on the discussion. Sorry if I'm a bit
behind in my
understanding of things.

Regarding the pyxform changes--

  1. Instead of the date, could we have the configuration
    parameter describe
    the specific change?

So instead of:

revision="2015-11-24"

we specify something like:

current_references_parent=True (default to False with the next
pyxform
version bump)

  1. Could we put this configuration into the xlsform's settings
    sheet? I
    think this would be easier to test and configure. We could also
    add a way to
    override the setting with an extra argument to the CLI or
    xls2json call.

On Tuesday, November 24, 2015 at 3:26:18 PM UTC-5, Mitch wrote:

I agree. Having just delved into the itemset code, there are a
number of
issues that would need to be cleaned up.

What I would propose is to leave JR / ODK Collect broken
as-is. This
will be future work.

The python code will then be the focus of the tweaks.

(1) change:

json_survey =

xls2json.workbook_to_json(json.loads(request.POST['workbookJson']),

form_name=form_name, warnings=warningsList)

to add an optional revision argument:

json_survey =

xls2json.workbook_to_json(json.loads(request.POST['workbookJson']),

form_name=form_name, warnings=warningsList,
revision='2015-11-24')

(2) change that implementation to:

(2a) store the revision string somewhere in the returned json
object.

(2b) expand ${fieldname} with the new relative path expansion
only if the
passed-in revision is >= '2015-11-24' ; otherwise, use the
absolute path
expansion.

(3) change:

create_survey_element_from_dict()

to emit the revision value as an attribute on the
element.

===============
The JR itemset bug will be fixed at a later date.

When that is fixed, we can introduce a new revision string for
that
behavior change.

i.e., once we have the revision string moving through the
system, we can
begin to support non-backward-compatible changes to behaviors
in XLSForm and
JR form processing.

For the time being, the ODK XLSForm converter will not supply
a revision
argument, and not make use of this pyxform enhancement. Once
the JR fix is
done, we will then support that new revision and the current
behavior,
defaulting to the new behavior, but supporting users in
generating the old
behavior.

Mitch

On Tue, Nov 24, 2015 at 8:44 AM, Martijn van de Rijdt < mar...@enketo.org> wrote:

I appreciate the creative solution. It does sound like it
will become a
bit of a headache to manage and explain to users though.
Whether it's worth
it depends on how many active forms are out there that rely
on the bug. We
should get Ona's and KoBo's input on this too, I think.

From Enketo's side, relative paths were never supported in
choice-filters (to my embarrassment and surprise - which led
to my
question). As a result, there is no significant issue with
backwards
compatibility in Enketo once it is fixed in JR. Not sure how
meaningful this
is to ODK Collect, but no Enketo user has ever reported this
lack of
support. I think to date it's been too hard to craft such
choice-filters.
Because of the way Enketo is integrated into KoBo, Ona,
Formhub, I'd be
surprised if there were a significant amount of those users
using relative
choice-filters.

I do indeed feel strongly about following the XForms 2.0 spec
here. I
think relative path support is going to be very useful in the
future, and I
agree that we need to look at the bigger picture and do this
correctly, so
it won't catch up with us.

On Tue, Nov 24, 2015 at 4:24 AM, Christopher Robert cro...@surveycto.com wrote:

Okay, well, this is turning into a bigger and bigger project
and we may
well miss the window for our 2.01 maintenance release
(and/or make the
changes so extensive and scary that we cannot in good
conscience include
them in a maintenance release).

However, your idea for making JR sensitive to the revision
value
emitted by pyxform opens a new possibility: we could release
the pyxform
change with 2.01 so that users can begin to benefit from the
relative paths
(relying on today's current() behavior), then assume that
when JR is "fixed"
it will only be fixed for XML emitted by newer pyxforms that
would have
updated their paths to account for the fix (using your
revision-value idea).
The primary drawback to this approach is that we would
basically break all
cascading-select filter expressions for SurveyCTO+Enketo
users (since
Martijn clearly wants to implement the "fixed" version of
current() rather
than the JavaRosa-compatible version of current()).

Meletis also found another issue relating to the evaluation
context in
jr:choice-name() (
https://github.com/opendatakit/opendatakit/issues/1180),
so we have that to contend with as well. If we can't safely
resolve the
context for jr:choice-name(), then that would present
another major drawback
and further lessen our enthusiasm for this entire adventure.

Best,

Chris

On Mon, Nov 23, 2015 at 5:45 PM Mitch Sundt < mitche...@gmail.com> wrote:

And, if it isn't clear, to preserve backward-compatible
operations,
XLSForm should not expand ${fieldname} with a relative path
unless the
revision value is the newer one; if the revision is not
specified, it would
emit absolute paths like it currently does.

I also switched to use a date value for the revision value
("2015-11-23") which allows simple lexical comparisons.

On Mon, Nov 23, 2015 at 1:03 PM, Mitch Sundt < mitche...@gmail.com> wrote:

w.r.t. backward compatibility

I too am concerned.

A proposed solution:

  1. Add a command line argument or other mechanism to
    request that
    XLSForm inject a 'revision="2015-11-23"' attribute into
    the element.
    e.g.,

This is used to alter the behavior of the XForms engine to
accommodate backward-compatible bug behaviors.

We can't identify the bugs (e.g,
'legacy="itemset-parent"'), since
that would break older forms run on the new ODK Collect
releases.
We have to advance the revision code as bugs are fixed.

I think an integer value is fine for this.

I don't anticipate making a lot of revisions, and I think
the
revisions would be sequentially applied (you can't
pick-and-choose what bugs
you include).

In the future, this would allow us to re-do the date-time
behavior.

  1. Modify the XLSForm tools (e.g., the online form
    generator) to
    enable the user to select what revision they want to emit
    (much like we did
    last year with the JR form eval choice setting within ODK
    Collect). We can
    default to emitting the latest revision if the user does
    not make a choice.

  2. Augment JR to look for this revision attribute and
    store/restore
    it in the FormDef binary object.

Based upon this value, alter the behavior of the JR itemset
evaluation to fix the path bug.

  1. Modify ODK Validate to scan for syntax that may be
    impacted by a
    revision (to advise whether to change something). Not
    sure how complex that
    might be. May not be always possible?

w.r.t. impacts of change

Thank you Meletis for researching when this change
occurred.

I will have to ponder what effect the parent/node change
would have
on itemset copying. Hadn't thought that there was any
relationship between
the two, but perhaps there is.

Yaw also reported that the Cascading Select itemsets on
opendatakit.appspot.com are not working with the 1.4.7
rev 1053 release (and
they work but give odd 'null' resolutions on 1057). This
is in a related
area of this same code, so I will hopefully gain a better
understanding of
dynamic itemsets as I uncover the causes of that problem.


Mitch

On Mon, Nov 23, 2015 at 5:59 AM, Christopher Robert cro...@surveycto.com wrote:

So just to be clear, the "fix" we're testing here reverts
the entire
context within that populateDynamicChoices() function, to
be the field
instead of the parent -- so the effects are potentially
much wider than just
what current() returns. Potentially anything and
everything that depends on
context can change now, since the context will shift from
parent to field.

This means that we revert a change from 5 years ago and
alter the
context from now on. Neither Meletis nor I feel terribly
confident that we
can entirely predict the consequences to backward
compatibility. We're
testing the things we can think to test, but honestly
it's XML users in the
broader ODK community who face the greatest risks here;
SurveyCTO users by
and large use XLSForm and thus the potential XPath syntax
consequences are
sharply limited.

Are you guys very sure that you want to take this path? I
can tell
you now that if we make this change and then there is
some outcry after the
next ODK Collect release, SurveyCTO Collect will very
likely stick with the
new change (since we will have changed pyxform and we'll
then have too many
backward-compatibility concerns to then reverse the
change).

Thanks,

Chris

On Mon, Nov 23, 2015 at 5:23 AM mel...@surveycto.com wrote:

Just for the history's sake, I found the time where the
behavior
changed. It was at March 11, 2010:

https://bitbucket.org/m.sundt/javarosa/commits/b41875b2c23c9a21a868a5d2dc0ffc1f6b5b0b65#Lcore/src/org/javarosa/core/model/FormDef.javaT635

Take a look only in the changes of the
"populateDynamicChoices"
method. The context reference used to be the "current"
field, but after the
changes above it started be the field's parent (by
adding getParent()).
Perhaps there was a reason for that change, but perhaps
not, but I think all
of us agree that this should be fixed, even after 5 and
a half years.

Meletis

On Monday, November 23, 2015 at 12:08:47 PM UTC+2, mel...@surveycto.com wrote:

Hi Mitch,

Please take a look at the code comment in line 31 here:

https://bitbucket.org/m.sundt/javarosa/src/bf17fb749b74e53915f086ca3827c3c6fe73fd07/core/src/org/javarosa/core/model/ItemsetBinding.java?at=default&fileviewer=file-view-default#ItemsetBinding.java-31

It clearly says: "ref of the control parent
(group/formdef) of
itemset question"

So, for some reason, they wanted it to point to the
parent. I also
see that when the dynamic choices are populated, the
evaluation context is
wrapped around that parent (contextRef object), that's
why current() refers
to the parent group when evaluating itemsets nodesets.

I will run a few tests by changing the contextRef field
value, I
just wanted to mention that in case you remember a
reason for that code
comment.

Meletis

On Friday, November 20, 2015 at 8:49:12 PM UTC+2, Mitch wrote:

Yes, I think it should be a one-line change; just
don't know
where.

Mitch

(I do jar-updates over Christmas / New-Years.
Tedious, but
doesn't require a whole lot of attention.)

On Fri, Nov 20, 2015 at 10:36 AM, < mel...@surveycto.com> wrote:

Hi Mitch,

Thank you for all the details, I guess I was missing
some of the
context of the original issue.

We will give it a shot to see if we can come up with
a few ideas
how to fix the evaluation content of the itemset
nodeset. I just hope it
will not be like last year with the "form processing
logic" in JR, otherwise
I believe that every Christmas will be a JR
Christmas. :slight_smile:

Meletis

On Friday, November 20, 2015 at 7:29:14 PM UTC+2, Mitch wrote:

The form was attached in one of my earlier replies.

On Fri, Nov 20, 2015 at 9:27 AM, Mitch Sundt mitche...@gmail.com wrote:

And you see this in constraint expressions where
you write:

. > 6

I.e., current()/. should refer to the field, and
not the group
containing the field, as it currently does when
evaluating expressions in
the itemset nodeset.

On Fri, Nov 20, 2015 at 9:26 AM, Mitch Sundt mitche...@gmail.com wrote:

The issue is that there is a discrepancy in the
meaning of
'.' across the different contexts of: calculate,
relevant, , and
these itemset nodeset expressions.

In other usages, you need to reference this other
field with
current()/../filterfield ; but in the nodeset
expression, you would
reference it with current()/./filterfield

I.e., the notion of '.' is different when used
within a
nodeset expression and when used within a
constraint expression or an
field.

See the attached NBiggestOfSet form. For the
label, it has:

    <select1

ref="/NBiggestOfSet/education/edu_level">

      <label>What is <output

value="current()/../referred_attendee_name"/>'s
level of education?
...

For a constraint, it has:
<bind

calculate="/NBiggestOfSet/attendee/name[position(..)=int(current()/../referred_attendee)]"

nodeset="/NBiggestOfSet/education/referred_attendee_name" readonly="true()"

type="string"/>

These each use the current()/../ to reference the
group (
/NBiggesetOfSet/education ).

But, within the nodeset definition (if this form
used one),
you would need to use
current()/./referred_attendee_name to reference a
field in that group.

On Fri, Nov 20, 2015 at 8:22 AM, < mel...@surveycto.com> wrote:

Hi Mitch,

I think I'm missing something but I think that
nothing is
broken and that the only issue is that the
expression
current()/../filterfield in your example above
should be just changed to
current()/filterfield. If you make that change,
everything should work
because it will be resolved to:

/data/group1/filterfield

Why should a "../" be present there? We have used
current()/somefield successfully in one of our
test forms and it works fine.
And I think it would also work great in your
example. Wouldn't it? Or am I
totally out of context?

Meletis

On Friday, November 20, 2015 at 5:35:16 PM UTC+2, Mitch wrote:

Hi Meletis,

That's not the issue --

The issue is that if you use a filter condition
in the
&g

--
Revolutionizing data collection since 2012.

Enketo https://enketo.org/ | LinkedIn
http://www.linkedin.com/company/enketo-llc | GitHub
https://github.com/enketo | Twitter
https://twitter.com/enketo | Blog http://blog.enketo.org/

--
You received this message because you are subscribed to the Google
Groups "ODK Developers" group.
To unsubscribe from this group and stop receiving emails from it, send
an email to opendatakit-developers+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--
Mitch Sundt
Software Engineer
University of Washington
mitchellsundt@gmail.com

--
You received this message because you are subscribed to the Google
Groups "ODK Developers" group.
To unsubscribe from this group and stop receiving emails from it, send
an email to opendatakit-developers+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google
Groups "ODK Developers" group.
To unsubscribe from this group and stop receiving emails from it, send
an email to opendatakit-developers+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--
Mitch Sundt
Software Engineer
University of Washington
mitchellsundt@gmail.com

--
You received this message because you are subscribed to the Google Groups
"ODK Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to opendatakit-developers+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

I am confused. Sorry if I missed this plan in the previous messages. Why
would you go ahead with adding the pyxform feature for the buggy predicate
behaviour? Right now we have almost no forms out there that rely on the
bug, i.e. almost no forms with the incorrect syntax. Why would we create
them?

Wouldn't it be better to add your highly-desirable feature only to forms
that have the current_always_field (or whatever it's called) flag set? For
the community-version of pyxform, I would advocate to do it like that.

··· On Thu, Dec 3, 2015 at 11:46 PM, Christopher Robert wrote:

FYI, I've just found a way to have pyxform only use current() when
replacing ${x} references that (a) are within choice_filter expressions and
(b) point to groups or fields within repeat groups. We're going to do a
bunch of testing to see how robust that change is (in terms of not breaking
any other cases and in terms of not imposing any performance costs). If it
looks solid, then it will become a candidate for inclusion in our 2.01
release.

That limits possible damage to cases that were already broken, and shifts
them to function based on the legacy JR support for current(). Our few
Enketo users won't benefit from the new ability to reference repeated
values in choice_filter expressions, but I think that's okay because
they're not losing anything. And any future fix to JR's current() behavior
will just need to be backward-compatible as discussed, based on flags in
the form model.

We'll be happy to share the pyxform changes with the community, if the
broader community would like to have this option to emit functional
choice_filter expressions for repeated fields. And the pyxform is easily
changed to emit XPaths compatible with the future-desired current()
behavior (though obviously the difficulty is in the backward compatibility,
as discussed at length).

Best,

Chris

On Fri, Dec 4, 2015 at 12:20 AM Christopher Robert crobert@surveycto.com wrote:

Thanks, Mitch. Some great ideas in there.

Best,

Chris

On Thu, Dec 3, 2015 at 12:52 PM Mitch Sundt mitchellsundt@gmail.com wrote:

It isn't that dire.

Not only do you need to then update 100% of your devices with a high
degree of certainty, but you also need to re-generate the XML for every
form on your server. And not even that is enough. Then you need to update
every form definition on every device.

=================
Individual Device Safeguards

Yes, all the devices need to be updated. And yes, this is the biggest
pain point. We CAN change the form download task so that it first scans
the form to detect unsupported revision tags, and refuses to install the
form definition on the device. And then throws up a Dialog warning the user
of the inappropriate form and prodding them to update ODK Collect. This
would preserve the older version of that individual form on that one
device. Survey CTO already added functionality to the download task to
ensure that all form attachments were downloaded before the form was copied
into the /sdcard/odk/forms directory and became 'live'. So this would be an
extra validity check before that wholesale move, allowing your non-upgraded
devices to stay 'in the past' w.r.t. form definitions they already have
until you upgrade the device.

This prevents catastrophic failure of an individual device, but does not
ensure complete migration of all the devices to the latest ODK Collect.

================
Device Fleet Management

ODK Aggregate, at least, does not have any mechanism to track all
devices and what their ODK Collect versions are. This is something that ODK
Manage was intended to do, but that never got implemented.

I haven't looked, but there are undoubtedly 3rd party tools that can
manage fleets of devices and communicate their configurations back to a
central location, and force-push updates onto individual devices. All the
anti-theft phone-lockout tools are basically doing that.

================
Form Revision Management

You do not need to upgrade all the forms at once.
Because the revision flags are in each form, you can keep running the
forms that require the bugs.
Even as you update your XLSForm, server and ODK Collect.

Depending upon how end-to-end your solution is, when running XLSForm to
update an existing form_id, you could access your live data collection
server to retrieve the existing form_id, and then, by default, apply the
same revision tags when making updates to that form_id. And only change
those revision tags through explicit user action.

That would allow users to easily edit the form to add another language
or fix a typo without much thought. It could also be expanded to run the
verification of identical data-models at that time (rather than later when
you try to upload the revision to the server) and to confirm that the form
version is appropriately larger than the existing form.

On Wed, Dec 2, 2015 at 4:37 PM, Christopher Robert < crobert@surveycto.com> wrote:

Thanks all, I appreciate all of the thinking going into this.

At the risk of thoroughly exhausting everybody's patience, I'm still
not sure that we have a strategy that can be described as reasonable from
an existing user's point of view. Let me explain why.

Say we release SurveyCTO 2.10 with three changes:

  1. We add the fix to JR and have it apply whenever the appropriate flag
    is set in the XForm model.

  2. We default all pyxform generation (which is basically invisible to
    the user in SurveyCTO) to include the appropriate flag so that
    newly-generated XML will be interpreted with the fixed JR.

  3. We default pyxform to output XPaths in choice-filter expressions
    using current() so that within-context repeat references finally work. This
    code, by default, relies on the fix (so paths are relative to the field,
    not the group). (It also changes ALL XPath expressions, not just those
    referencing within-repeat values, because we're not clever enough to be
    able to distinguish the different cases in that python code.)

Now say you're an upgrading user and you upgraded because SurveyCTO
2.10 also included a beautiful and astonishingly powerful web-based form
designer. You upgraded to get this new designer and you love it and now you
upload a new or revised form. That form happens to have a choice-filter
expression. Now here's where things get ugly...

That form won't work on your SurveyCTO 2.01, 2.00, 1.402, etc. devices.
Probably there's some nasty XPath parsing error when you get to where the
choice filter kicks in, maybe it even crashes your Collect. That form might
have worked the day before, and maybe all you changed was fixing a typo in
a label, but now suddenly it doesn't work on all of these devices.

So here's where we say, okay, the user has two choices: (1) upgrade all
devices to the 2.10 Collect, or (2) find and set the "stay in the past"
option. The second option is good in principle -- because it provides a
kind of quick save -- but then what does it take to ever migrate away from
that "stay in the past" setting? Not only do you need to then update 100%
of your devices with a high degree of certainty, but you also need to
re-generate the XML for every form on your server. And not even that is
enough. Then you need to update every form definition on every device.

And if you miss a few devices and leave them on the old version, or
miss a few form definitions here or there? Cryptic errors, inability to
finalize forms, confusion, anger, frantic calls to support, etc., etc.

You don't need to be a human-centered design expert to see that this is
kind of a disaster from a user-experience point of view. And, for us, this
would be multiplied across hundreds of migrating users, essentially eating
a lot of our 2016 and causing a tremendous amount of ill will.

The only upgrade path for existing users that makes any sense is one
where we basically emit the "stay in the past" flag always for everyone.
You can all turn your noses up at the idea of indefinite reliance on what
was initially a bug, and I get it. But at least here at SurveyCTO, we're
not going to put our users through the kind of migration path described
above.

If relying on the bug is just too unpalatable, then ultimately I don't
think that we can change pyxform to emit XPaths using current(). It's just
too fraught and we don't have a good way to manage the upgrade path without
torturing existing users. So fine, our awesome pyxform fix to magically
make within-repeat references start working is just not destined to see the
light of day. Rather, we need to make pyxform smart enough to only use
current() for paths to repeated values (which never worked in old JR's
anyway). Or we need to just provide a new solution for referencing repeated
values without current() (perhaps a new custom JR function, which everybody
loves).

And in that case, sure, I'm much more relaxed about the strategies for
upgrading users since very few are likely to be using current(). I won't
want to overly torture those users, but I won't lose a ton of sleep over
them since even users who use current() now are unlikely to use it in a lot
of forms.

I'll see if we can figure out how to make the pyxform change in such a
way that it only changes the XPaths for fields within repeat groups. But I
might also see if we can instead find a totally different way to solve or
work around this problem. Or, I'll see how I feel about pushing this off
for another year. Fixing choice filters to be able to work will within
repeat groups was not meant to be a huge project.

Thanks again,

Chris

On Wed, Dec 2, 2015 at 2:48 PM Mitch Sundt mitchellsundt@gmail.com wrote:

w.r.t. moving forward...

Even if this particular change impacts a very small number of forms,
we should figure out how to implement non-backward-compatible changes so
that they are safe moving forward. We have no visibility into who uses ODK
Collect and ODK Aggregate and no reliable way to notify them of changes.

This could be a huge deal when or if we ever revise the JR engine to
properly handle timezones, dates and times, and date and time arithmetic.

Or if we find a need to change XLSForm to emit XForms features or
XPath functions that currently don't work (e.g., to use [1] or position(),
or XForms events, for example).

Note that these non-backward-compatible changes extend to not just
XLSForm and ODK Collect, but also to servers. You can use the same revision
tag to indicate new or different processing to be taken on the server.

=============
Managing this change requires:

I believe that this requires:

(1) the XML form definition is the place to define the
feature-enabling-labels that the form requires of the device and server
processing steps.

(2) when the JR engine or any tool implements non-backward-compatible
changes, it only activates them when it detects the presence of those
feature-enabling-label in the XML form definition. Omitting that
feature-enabling-label is a directive to "stay in the past".

(3) ODK Collect and other tools processing the XML form definition or
the XML submission documents should warn the user (e.g., pop up a Toast) if
they are running a version that does not recognize the
feature-enabling-labels in the form's revision tag (i.e., they are given a
form that indicates an advanced feature that they are unaware of) -- this
notification informs the user that they should upgrade that tool to
correctly process the directive. Alternatively, the tool could reject the
form, Force Close or abort, which is the current behavior when ODK
Aggregate or ODK Collect gets an XForms definition that uses a new,
unrecognized, function, etc. ;-<

=============
The stable upgrade path is to:

(1) update your form distribution and data aggregation server(s),
(2) update your collection devices (or collection processing server --
e.g., Enketo), and
(3) only then update the form generation tool to emit forms with the
non-backward-compatible change.

=============
Specifying the Revision settings:

I believe the XLS file should not contain a setting to select or
specify the revision
.

I believe this should be specified at the time the XLSForm converter
is executed on the source XLS file.
I.e., the web UI for these tools should have a spinner or other UI
element that enables users to choose to "stay in the past"

This forces users to actively stay in the past.

The ODK site will probably achieve this with a drop-down that allows
the user to specify, e.g.,

"full current functionality"
"generate a form definition compatible with ODK Collect 1.4.7 and
earlier"

To allow all groups to independently advance the functionality within
their tool families, the combination of features that each of these
drop-down choices corresponds to should be documented by those groups.

I.e., I see a future where a form generated on Ona for Enketo may not
be compatible with stock ODK Collect unless an appropriate compatibility
drop-down is selected.

This could allow, for example, Survey CTO to implement and recoup
costs for JR date-time fixes or other XForms extensions within their
ecosystem before releasing that fix back to the wider community.

I do not see any value in offering a user the choice of which features
are enabled or disabled. I suppose some sites could offer that, but it
seems overly complex. Users really just need a way to specify compatibility
to whatever older version of the tools they are using.

==============
XLSForm functionality

I am fine with the XLSForm generator taking a comma-separated list of
features to enable, and passing that list through via the revision tag to
the the XML form definition.

Upon reflection, I think this is the only way to manage multiple
independent parallel contributions to the various codebases. It allows
different groups to add and enable different functionalities at different
rates.

We just need to maintain a global list of all of the feature labels
that every group is using so that we don't choose names that collide.

And, some of these feature labels may simply pass through the XLSForm
generator and on to the downstream tools. e.g., date-time fixes might not
have any impact on XLSForm, but would alter how the JR engine processed the
form.

On Wed, Dec 2, 2015 at 10:25 AM, Martijn van de Rijdt < martijn@enketo.org> wrote:

Ok, I was just wondering if it's really worth the effort. KoBo found
3 forms using current(). Not sure how many are actively used still. For
them it would seem to me easier to just contact that/those user(s)
individually to come up with a plan.

I can live with the XForm attribute and XLSForm setting addition.

Chris, to respond to your question about adding support for the bug
to Enketo. I don't currently see a reason to do so, as no Enketo user has
ever relied on the bug and I highly doubt any will in the future. However,
I don't think this should be relevant to this discussion. We're all in
agreement that we want the community to move forward with the correct
syntax.

On Tuesday, December 1, 2015 at 4:58:45 PM UTC-7, Christopher Robert wrote:

We have individual users with 500+ devices in the field. Updating
devices -- or even keeping track of which devices are on which versions --
is a major undertaking. In total, we're talking many thousands of devices
spread across many countries. The problem is not a small one, and our
commitment to backward compatibility is one that our users not only
appreciate but expect.

ODK/SurveyCTO Collect continues to use older versions of forms to
view, edit, or finalize submissions begun under those older versions, even
when newer versions exist. So the backward-compatibility challenge is a
formidable one. If you change the way JavaRosa interprets expressions in
the kind of blanket way that you suggest, then users will be faced with the
impossible choice of running software that fails to function properly with
existing, in-progress, or old submissions, or running software that fails
to function properly with newer submissions. Multiply the difficulty of the
choice and the potential confusion of XPath parsing errors times
however-many devices there are in the field.

No, this is a legitimately difficult situation that requires a
tremendous amount of care. Changing how XPath expressions are interpreted
is a big deal for the Collect world, though I can see why it would be
considerably less so in the Enketo world.

Now, I will admit that there are bound to be very few users who
already use current() in choice filters. Mitch suggested the approach a few
times in the public forum, but even I failed to get it to work the first
several times I tried to follow his advice. So chances are that not many
projects are using current() today. (I know of 1-2, maybe there are 8-10
total. That's just a guess.)

Enketo aside, we could -- today! -- allow people to reference
in-context repeated values by exploiting the current JR support for
current(). For some users/projects, there would be an immediate benefit.
But obviously that relies on "the bug" and so it would expand the potential
need for backward compatibility. And it would break choice-filter
compatibility between Collect and Enketo (say, for SurveyCTO users at
least).

It's easy to say "just fix the bug and then this problem goes away,"
but the fact is that we don't quite know how to fix it without breaking
other things; we have a candidate for a fix, but it's seriously scary. It
may take time. And meanwhile, we have already gone multiple years with
users waiting to be able to reference within-context repeated values in
choice filters. If a safe fix drags on for another year or two -- either
because it's just too hard or because nobody quite has the time -- is that
just the way it goes? I don't know. It's not obvious to me that breaking
compatibility with Enketo and relying on current behavior isn't the
welfare-maximizing choice.

In the end, I believe that Mitch's suggestion re: compatibility is
maybe necessary (storing in the XML some indication of how current() in
choice filters should be interpreted). And if we totally screwed up the fix
and broke a bunch of use cases, at least it might offer some escape for
affected users (if we allowed an override on the settings sheet).

So no, I don't agree that it's as simple as "fix the bug and make
people upgrade."

Chris

On Tue, Dec 1, 2015 at 5:11 PM Martijn van de Rijdt < mar...@enketo.org> wrote:

I can't help thinking that we're overthinking this
backwards-compatibility issue.

Ideally we have old clients that have the bug and new clients that
do not have the bug. Isn't that the case with any other major bug fix? If
an old form stops working in the new version of ODK Collect, the survey
owner will either have to upload a new version of the form or the user will
have to downgrade ODK Collect. Surely this will be a problem for some
users, but for how many? A small fraction of one percent seems like a
reasonable guess.

Maybe the backwards-compatibility should be addressed just by
warning users, as Yaw suggested way above.

Any 'side-effects' of fixing the bug, like with creating itemsets,
can be tested for, I presume, before deploying the fix.

One way to detect at least a significant portion of the current
current() users, is to do a regex search for any usage of the
'current()/path/to/node' where current()/ is not followed by '../'. That
usage is specific to this bug to refer to a sibling node and will not occur
any more after the bug is fixed (because a question cannot have children).

On Mon, Nov 30, 2015 at 11:43 AM, Christopher Robert < cro...@surveycto.com> wrote:

I wasn't going to suggest adding the different parameters to any
of the XLSForm templates -- so they wouldn't become cluttered with this
stuff. Really, it would just be an emergency option for somebody requiring
different compatibility for some component that was out of sync with their
other components. Ideally, nobody would actually have to use the settings
sheet to override any of these options.

To be specific, my proposal would be that everybody
support current_always_field=false today, and that this be how pyxform
today defaults. Ideally, this would include Enketo, as Collect and
SurveyCTO's Web Collect (and anything else using the ODK JavaRosa) already
support current_always_field=false as their current behavior. Then, we
figure out how to get current_always_field=true to work in JavaRosa and
phase that in, eventually changing the pyxform default to that once we
think that enough components support it.

We have thousands of devices out there using older Collect
versions, even well after their servers have been updated... so my guess is
that it's some of those users who will end up having to use the
settings-sheet option to force current_always_field=false after we've
changed the default to true.

Does all of that make sense?

Chris

On Sun, Nov 29, 2015 at 6:48 PM Yaw Anokwa yan...@nafundi.com wrote:

All this sounds fine to me, but wanted to raise some potential
issues.
Discrete parameters also means folks can pick and choose what
behavior
they want and it's hard to move the entire community forward that
way.
Also, once you get past two or three, you end up with a settings
sheet
of lots of flags to toggle and that isn't a great user experience.

Yaw

Need ODK consultants? Nafundi provides form design, server setup,
in-field training, and software development for ODK. Go to
https://nafundi.com to get started.

On Sun, Nov 29, 2015 at 6:47 PM, Christopher Robert cro...@surveycto.com wrote:

Having a number of discrete parameters that are overridable on
the settings
sheet makes sense to me (like "current_references_parent" or
perhaps
"current_always_field"). We could then add model parameters and
change the
pyxform default over time, but users could always override on
their settings
sheets as needed for specific compatibility.

For example, our SurveyCTO pyxform for 2.10 (we've given up on
2.01 now)
could default to current_always_field=false, which would mean
that JR would
be backward-compatible and pyxform would emit relative paths
for the
choice_filter column assuming that current() references the
parent. This
would break all choice_filters in Enketo, but our Enketo users
could set
current_always_field=true in their settings sheets.

For that to work, though, we'd also have to add support for
current_always_field=true in JR. We could do that as Meletis
has proposed,
which may have other side effects -- we'll have to see (e.g.,
for itemset
copying).

Alternatively, we could try to convince Martijn to support
current_always_field=false, which is likely trivial. If he
would do that,
then we could push JR's support for current_always_field=true
to when we
could figure out how to do it safely. Also, it would mean
producing a
version of Enketo that supports relative paths in choice
filters in a manner
compatible with ODK Collect. Ona, Kobo, and other Enketo users
could
rejoice, able to produce forms that use current() (like those
that reference
same-repeat values) in their choice_filters. That boon to users
has to be
worth something, even if a principle is slightly violated
(supporting
something outside the XPath spec).

Chris

On Tue, Nov 24, 2015 at 8:43 PM Alex Dorey dore...@gmail.com wrote:

Hi, Just catching up on the discussion. Sorry if I'm a bit
behind in my
understanding of things.

Regarding the pyxform changes--

  1. Instead of the date, could we have the configuration
    parameter describe
    the specific change?

So instead of:

revision="2015-11-24"

we specify something like:

current_references_parent=True (default to False with the
next pyxform
version bump)

  1. Could we put this configuration into the xlsform's settings
    sheet? I
    think this would be easier to test and configure. We could
    also add a way to
    override the setting with an extra argument to the CLI or
    xls2json call.

On Tuesday, November 24, 2015 at 3:26:18 PM UTC-5, Mitch wrote:

I agree. Having just delved into the itemset code, there are
a number of
issues that would need to be cleaned up.

What I would propose is to leave JR / ODK Collect broken
as-is. This
will be future work.

The python code will then be the focus of the tweaks.

(1) change:

json_survey =

xls2json.workbook_to_json(json.loads(request.POST['workbookJson']),

form_name=form_name, warnings=warningsList)

to add an optional revision argument:

json_survey =

xls2json.workbook_to_json(json.loads(request.POST['workbookJson']),

form_name=form_name, warnings=warningsList,
revision='2015-11-24')

(2) change that implementation to:

(2a) store the revision string somewhere in the returned json
object.

(2b) expand ${fieldname} with the new relative path expansion
only if the
passed-in revision is >= '2015-11-24' ; otherwise, use the
absolute path
expansion.

(3) change:

create_survey_element_from_dict()

to emit the revision value as an attribute on the
element.

===============
The JR itemset bug will be fixed at a later date.

When that is fixed, we can introduce a new revision string
for that
behavior change.

i.e., once we have the revision string moving through the
system, we can
begin to support non-backward-compatible changes to behaviors
in XLSForm and
JR form processing.

For the time being, the ODK XLSForm converter will not supply
a revision
argument, and not make use of this pyxform enhancement. Once
the JR fix is
done, we will then support that new revision and the current
behavior,
defaulting to the new behavior, but supporting users in
generating the old
behavior.

Mitch

On Tue, Nov 24, 2015 at 8:44 AM, Martijn van de Rijdt < mar...@enketo.org> wrote:

I appreciate the creative solution. It does sound like it
will become a
bit of a headache to manage and explain to users though.
Whether it's worth
it depends on how many active forms are out there that rely
on the bug. We
should get Ona's and KoBo's input on this too, I think.

From Enketo's side, relative paths were never supported in
choice-filters (to my embarrassment and surprise - which led
to my
question). As a result, there is no significant issue with
backwards
compatibility in Enketo once it is fixed in JR. Not sure how
meaningful this
is to ODK Collect, but no Enketo user has ever reported this
lack of
support. I think to date it's been too hard to craft such
choice-filters.
Because of the way Enketo is integrated into KoBo, Ona,
Formhub, I'd be
surprised if there were a significant amount of those users
using relative
choice-filters.

I do indeed feel strongly about following the XForms 2.0
spec here. I
think relative path support is going to be very useful in
the future, and I
agree that we need to look at the bigger picture and do this
correctly, so
it won't catch up with us.

On Tue, Nov 24, 2015 at 4:24 AM, Christopher Robert cro...@surveycto.com wrote:

Okay, well, this is turning into a bigger and bigger
project and we may
well miss the window for our 2.01 maintenance release
(and/or make the
changes so extensive and scary that we cannot in good
conscience include
them in a maintenance release).

However, your idea for making JR sensitive to the revision
value
emitted by pyxform opens a new possibility: we could
release the pyxform
change with 2.01 so that users can begin to benefit from
the relative paths
(relying on today's current() behavior), then assume that
when JR is "fixed"
it will only be fixed for XML emitted by newer pyxforms
that would have
updated their paths to account for the fix (using your
revision-value idea).
The primary drawback to this approach is that we would
basically break all
cascading-select filter expressions for SurveyCTO+Enketo
users (since
Martijn clearly wants to implement the "fixed" version of
current() rather
than the JavaRosa-compatible version of current()).

Meletis also found another issue relating to the evaluation
context in
jr:choice-name() (
https://github.com/opendatakit/opendatakit/issues/1180),
so we have that to contend with as well. If we can't safely
resolve the
context for jr:choice-name(), then that would present
another major drawback
and further lessen our enthusiasm for this entire adventure.

Best,

Chris

On Mon, Nov 23, 2015 at 5:45 PM Mitch Sundt < mitche...@gmail.com> wrote:

And, if it isn't clear, to preserve backward-compatible
operations,
XLSForm should not expand ${fieldname} with a relative
path unless the
revision value is the newer one; if the revision is not
specified, it would
emit absolute paths like it currently does.

I also switched to use a date value for the revision value
("2015-11-23") which allows simple lexical comparisons.

On Mon, Nov 23, 2015 at 1:03 PM, Mitch Sundt < mitche...@gmail.com> wrote:

w.r.t. backward compatibility

I too am concerned.

A proposed solution:

  1. Add a command line argument or other mechanism to
    request that
    XLSForm inject a 'revision="2015-11-23"' attribute into
    the element.
    e.g.,

This is used to alter the behavior of the XForms engine to
accommodate backward-compatible bug behaviors.

We can't identify the bugs (e.g,
'legacy="itemset-parent"'), since
that would break older forms run on the new ODK Collect
releases.
We have to advance the revision code as bugs are fixed.

I think an integer value is fine for this.

I don't anticipate making a lot of revisions, and I think
the
revisions would be sequentially applied (you can't
pick-and-choose what bugs
you include).

In the future, this would allow us to re-do the date-time
behavior.

  1. Modify the XLSForm tools (e.g., the online form
    generator) to
    enable the user to select what revision they want to emit
    (much like we did
    last year with the JR form eval choice setting within ODK
    Collect). We can
    default to emitting the latest revision if the user does
    not make a choice.

  2. Augment JR to look for this revision attribute and
    store/restore
    it in the FormDef binary object.

Based upon this value, alter the behavior of the JR
itemset
evaluation to fix the path bug.

  1. Modify ODK Validate to scan for syntax that may be
    impacted by a
    revision (to advise whether to change something). Not
    sure how complex that
    might be. May not be always possible?

w.r.t. impacts of change

Thank you Meletis for researching when this change
occurred.

I will have to ponder what effect the parent/node change
would have
on itemset copying. Hadn't thought that there was any
relationship between
the two, but perhaps there is.

Yaw also reported that the Cascading Select itemsets on
opendatakit.appspot.com are not working with the 1.4.7
rev 1053 release (and
they work but give odd 'null' resolutions on 1057). This
is in a related
area of this same code, so I will hopefully gain a better
understanding of
dynamic itemsets as I uncover the causes of that problem.


Mitch

On Mon, Nov 23, 2015 at 5:59 AM, Christopher Robert cro...@surveycto.com wrote:

So just to be clear, the "fix" we're testing here
reverts the entire
context within that populateDynamicChoices() function,
to be the field
instead of the parent -- so the effects are potentially
much wider than just
what current() returns. Potentially anything and
everything that depends on
context can change now, since the context will shift
from parent to field.

This means that we revert a change from 5 years ago and
alter the
context from now on. Neither Meletis nor I feel terribly
confident that we
can entirely predict the consequences to backward
compatibility. We're
testing the things we can think to test, but honestly
it's XML users in the
broader ODK community who face the greatest risks here;
SurveyCTO users by
and large use XLSForm and thus the potential XPath
syntax consequences are
sharply limited.

Are you guys very sure that you want to take this path?
I can tell
you now that if we make this change and then there is
some outcry after the
next ODK Collect release, SurveyCTO Collect will very
likely stick with the
new change (since we will have changed pyxform and we'll
then have too many
backward-compatibility concerns to then reverse the
change).

Thanks,

Chris

On Mon, Nov 23, 2015 at 5:23 AM mel...@surveycto.com wrote:

Just for the history's sake, I found the time where the
behavior
changed. It was at March 11, 2010:

https://bitbucket.org/m.sundt/javarosa/commits/b41875b2c23c9a21a868a5d2dc0ffc1f6b5b0b65#Lcore/src/org/javarosa/core/model/FormDef.javaT635

Take a look only in the changes of the
"populateDynamicChoices"
method. The context reference used to be the "current"
field, but after the
changes above it started be the field's parent (by
adding getParent()).
Perhaps there was a reason for that change, but perhaps
not, but I think all
of us agree that this should be fixed, even after 5 and
a half years.

Meletis

On Monday, November 23, 2015 at 12:08:47 PM UTC+2, mel...@surveycto.com wrote:

Hi Mitch,

Please take a look at the code comment in line 31 here:

https://bitbucket.org/m.sundt/javarosa/src/bf17fb749b74e53915f086ca3827c3c6fe73fd07/core/src/org/javarosa/core/model/ItemsetBinding.java?at=default&fileviewer=file-view-default#ItemsetBinding.java-31

It clearly says: "ref of the control parent
(group/formdef) of
itemset question"

So, for some reason, they wanted it to point to the
parent. I also
see that when the dynamic choices are populated, the
evaluation context is
wrapped around that parent (contextRef object), that's
why current() refers
to the parent group when evaluating itemsets nodesets.

I will run a few tests by changing the contextRef
field value, I
just wanted to mention that in case you remember a
reason for that code
comment.

Meletis

On Friday, November 20, 2015 at 8:49:12 PM UTC+2, Mitch wrote:

Yes, I think it should be a one-line change; just
don't know
where.

Mitch

(I do jar-updates over Christmas / New-Years.
Tedious, but
doesn't require a whole lot of attention.)

On Fri, Nov 20, 2015 at 10:36 AM, < mel...@surveycto.com> wrote:

Hi Mitch,

Thank you for all the details, I guess I was missing
some of the
context of the original issue.

We will give it a shot to see if we can come up with
a few ideas
how to fix the evaluation content of the itemset
nodeset. I just hope it
will not be like last year with the "form processing
logic" in JR, otherwise
I believe that every Christmas will be a JR
Christmas. :slight_smile:

Meletis

On Friday, November 20, 2015 at 7:29:14 PM UTC+2, Mitch wrote:

The form was attached in one of my earlier replies.

On Fri, Nov 20, 2015 at 9:27 AM, Mitch Sundt mitche...@gmail.com wrote:

And you see this in constraint expressions where
you write:

. > 6

I.e., current()/. should refer to the field, and
not the group
containing the field, as it currently does when
evaluating expressions in
the itemset nodeset.

On Fri, Nov 20, 2015 at 9:26 AM, Mitch Sundt mitche...@gmail.com wrote:

The issue is that there is a discrepancy in the
meaning of
'.' across the different contexts of: calculate,
relevant, , and
these itemset nodeset expressions.

In other usages, you need to reference this other
field with
current()/../filterfield ; but in the nodeset
expression, you would
reference it with current()/./filterfield

I.e., the notion of '.' is different when used
within a
nodeset expression and when used within a
constraint expression or an
field.

See the attached NBiggestOfSet form. For the
label, it has:

    <select1

ref="/NBiggestOfSet/education/edu_level">

      <label>What is <output

value="current()/../referred_attendee_name"/>'s
level of education?
...

For a constraint, it has:
<bind

calculate="/NBiggestOfSet/attendee/name[position(..)=int(current()/../referred_attendee)]"

nodeset="/NBiggestOfSet/education/referred_attendee_name" readonly="true()"

type="string"/>

These each use the current()/../ to reference the
group (
/NBiggesetOfSet/education ).

But, within the nodeset definition (if this form
used one),
you would need to use
current()/./referred_attendee_name to reference a
field in that group.

On Fri, Nov 20, 2015 at 8:22 AM, < mel...@surveycto.com> wrote:

Hi Mitch,

I think I'm missing something but I think that
nothing is
broken and that the only issue is that the
expression
current()/../filterfield in your example above
should be just changed to
current()/filterfield. If you make that change,
everything should work
because it will be resolved to:

/data/group1/filterfield

Why should a "../" be present there? We have used
current()/somefield successfully in one of our
test forms and it works fine.
And I think it would also work great in your
example. Wouldn't it? Or am I
totally out of context?

Meletis

On Friday, November 20, 2015 at 5:35:16 PM UTC+2, Mitch wrote:

Hi Meletis,

That's not the issue --

The issue is that if you use a filter condition
in the
&g

--
Revolutionizing data collection since 2012.

Enketo https://enketo.org/ | LinkedIn
http://www.linkedin.com/company/enketo-llc | GitHub
https://github.com/enketo | Twitter
https://twitter.com/enketo | Blog http://blog.enketo.org/

--
You received this message because you are subscribed to the Google
Groups "ODK Developers" group.
To unsubscribe from this group and stop receiving emails from it,
send an email to opendatakit-developers+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--
Mitch Sundt
Software Engineer
University of Washington
mitchellsundt@gmail.com

--
You received this message because you are subscribed to the Google
Groups "ODK Developers" group.
To unsubscribe from this group and stop receiving emails from it, send
an email to opendatakit-developers+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google
Groups "ODK Developers" group.
To unsubscribe from this group and stop receiving emails from it, send
an email to opendatakit-developers+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--
Mitch Sundt
Software Engineer
University of Washington
mitchellsundt@gmail.com

--
You received this message because you are subscribed to the Google
Groups "ODK Developers" group.
To unsubscribe from this group and stop receiving emails from it, send
an email to opendatakit-developers+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to a topic in the
Google Groups "ODK Developers" group.
To unsubscribe from this topic, visit
https://groups.google.com/d/topic/opendatakit-developers/nN8HTA5pUIM/unsubscribe
.
To unsubscribe from this group and all its topics, send an email to
opendatakit-developers+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--

Revolutionizing data collection since 2012.

Enketo https://enketo.org/ | LinkedIn
http://www.linkedin.com/company/enketo-llc | GitHub
https://github.com/enketo | Twitter https://twitter.com/enketo
| Blog http://blog.enketo.org/

Mitch, Thanks for your detailed proposal.

I think I agree with not making any of these flags a setting in XLSForm.

I just want to suggest/confirm we don't use this for features, but use it
only for backward-incompatible changes in evaluating XPath. (I'm
personally super excited about one day using it to eliminate this XForms
deviation
http://opendatakit.github.io/odk-xform-spec/#a-big-deviation-with-xforms!)
With respect to features (like geoshape support), we should put the
responsibility on the survey manager to figure out which clients to use.

··· On Fri, Dec 4, 2015 at 9:39 AM, Martijn van de Rijdt wrote:

I am confused. Sorry if I missed this plan in the previous messages. Why
would you go ahead with adding the pyxform feature for the buggy predicate
behaviour? Right now we have almost no forms out there that rely on the
bug, i.e. almost no forms with the incorrect syntax. Why would we
create them?

Wouldn't it be better to add your highly-desirable feature only to
forms that have the current_always_field (or whatever it's called) flag
set? For the community-version of pyxform, I would advocate to do it like
that.

On Thu, Dec 3, 2015 at 11:46 PM, Christopher Robert <crobert@surveycto.com wrote:

FYI, I've just found a way to have pyxform only use current() when
replacing ${x} references that (a) are within choice_filter expressions and
(b) point to groups or fields within repeat groups. We're going to do a
bunch of testing to see how robust that change is (in terms of not breaking
any other cases and in terms of not imposing any performance costs). If it
looks solid, then it will become a candidate for inclusion in our 2.01
release.

That limits possible damage to cases that were already broken, and shifts
them to function based on the legacy JR support for current(). Our few
Enketo users won't benefit from the new ability to reference repeated
values in choice_filter expressions, but I think that's okay because
they're not losing anything. And any future fix to JR's current() behavior
will just need to be backward-compatible as discussed, based on flags in
the form model.

We'll be happy to share the pyxform changes with the community, if the
broader community would like to have this option to emit functional
choice_filter expressions for repeated fields. And the pyxform is easily
changed to emit XPaths compatible with the future-desired current()
behavior (though obviously the difficulty is in the backward compatibility,
as discussed at length).

Best,

Chris

On Fri, Dec 4, 2015 at 12:20 AM Christopher Robert crobert@surveycto.com wrote:

Thanks, Mitch. Some great ideas in there.

Best,

Chris

On Thu, Dec 3, 2015 at 12:52 PM Mitch Sundt mitchellsundt@gmail.com wrote:

It isn't that dire.

Not only do you need to then update 100% of your devices with a high
degree of certainty, but you also need to re-generate the XML for every
form on your server. And not even that is enough. Then you need to update
every form definition on every device.

=================
Individual Device Safeguards

Yes, all the devices need to be updated. And yes, this is the biggest
pain point. We CAN change the form download task so that it first scans
the form to detect unsupported revision tags, and refuses to install the
form definition on the device. And then throws up a Dialog warning the user
of the inappropriate form and prodding them to update ODK Collect. This
would preserve the older version of that individual form on that one
device. Survey CTO already added functionality to the download task to
ensure that all form attachments were downloaded before the form was copied
into the /sdcard/odk/forms directory and became 'live'. So this would be an
extra validity check before that wholesale move, allowing your non-upgraded
devices to stay 'in the past' w.r.t. form definitions they already have
until you upgrade the device.

This prevents catastrophic failure of an individual device, but does
not ensure complete migration of all the devices to the latest ODK Collect.

================
Device Fleet Management

ODK Aggregate, at least, does not have any mechanism to track all
devices and what their ODK Collect versions are. This is something that ODK
Manage was intended to do, but that never got implemented.

I haven't looked, but there are undoubtedly 3rd party tools that can
manage fleets of devices and communicate their configurations back to a
central location, and force-push updates onto individual devices. All the
anti-theft phone-lockout tools are basically doing that.

================
Form Revision Management

You do not need to upgrade all the forms at once.
Because the revision flags are in each form, you can keep running the
forms that require the bugs.
Even as you update your XLSForm, server and ODK Collect.

Depending upon how end-to-end your solution is, when running XLSForm to
update an existing form_id, you could access your live data collection
server to retrieve the existing form_id, and then, by default, apply the
same revision tags when making updates to that form_id. And only change
those revision tags through explicit user action.

That would allow users to easily edit the form to add another language
or fix a typo without much thought. It could also be expanded to run the
verification of identical data-models at that time (rather than later when
you try to upload the revision to the server) and to confirm that the form
version is appropriately larger than the existing form.

On Wed, Dec 2, 2015 at 4:37 PM, Christopher Robert < crobert@surveycto.com> wrote:

Thanks all, I appreciate all of the thinking going into this.

At the risk of thoroughly exhausting everybody's patience, I'm still
not sure that we have a strategy that can be described as reasonable from
an existing user's point of view. Let me explain why.

Say we release SurveyCTO 2.10 with three changes:

  1. We add the fix to JR and have it apply whenever the appropriate
    flag is set in the XForm model.

  2. We default all pyxform generation (which is basically invisible to
    the user in SurveyCTO) to include the appropriate flag so that
    newly-generated XML will be interpreted with the fixed JR.

  3. We default pyxform to output XPaths in choice-filter expressions
    using current() so that within-context repeat references finally work. This
    code, by default, relies on the fix (so paths are relative to the field,
    not the group). (It also changes ALL XPath expressions, not just those
    referencing within-repeat values, because we're not clever enough to be
    able to distinguish the different cases in that python code.)

Now say you're an upgrading user and you upgraded because SurveyCTO
2.10 also included a beautiful and astonishingly powerful web-based form
designer. You upgraded to get this new designer and you love it and now you
upload a new or revised form. That form happens to have a choice-filter
expression. Now here's where things get ugly...

That form won't work on your SurveyCTO 2.01, 2.00, 1.402, etc.
devices. Probably there's some nasty XPath parsing error when you get to
where the choice filter kicks in, maybe it even crashes your Collect. That
form might have worked the day before, and maybe all you changed was fixing
a typo in a label, but now suddenly it doesn't work on all of these devices.

So here's where we say, okay, the user has two choices: (1) upgrade
all devices to the 2.10 Collect, or (2) find and set the "stay in the past"
option. The second option is good in principle -- because it provides a
kind of quick save -- but then what does it take to ever migrate away from
that "stay in the past" setting? Not only do you need to then update 100%
of your devices with a high degree of certainty, but you also need to
re-generate the XML for every form on your server. And not even that is
enough. Then you need to update every form definition on every device.

And if you miss a few devices and leave them on the old version, or
miss a few form definitions here or there? Cryptic errors, inability to
finalize forms, confusion, anger, frantic calls to support, etc., etc.

You don't need to be a human-centered design expert to see that this
is kind of a disaster from a user-experience point of view. And, for us,
this would be multiplied across hundreds of migrating users, essentially
eating a lot of our 2016 and causing a tremendous amount of ill will.

The only upgrade path for existing users that makes any sense is one
where we basically emit the "stay in the past" flag always for everyone.
You can all turn your noses up at the idea of indefinite reliance on what
was initially a bug, and I get it. But at least here at SurveyCTO, we're
not going to put our users through the kind of migration path described
above.

If relying on the bug is just too unpalatable, then ultimately I don't
think that we can change pyxform to emit XPaths using current(). It's just
too fraught and we don't have a good way to manage the upgrade path without
torturing existing users. So fine, our awesome pyxform fix to magically
make within-repeat references start working is just not destined to see the
light of day. Rather, we need to make pyxform smart enough to only use
current() for paths to repeated values (which never worked in old JR's
anyway). Or we need to just provide a new solution for referencing repeated
values without current() (perhaps a new custom JR function, which everybody
loves).

And in that case, sure, I'm much more relaxed about the strategies for
upgrading users since very few are likely to be using current(). I won't
want to overly torture those users, but I won't lose a ton of sleep over
them since even users who use current() now are unlikely to use it in a lot
of forms.

I'll see if we can figure out how to make the pyxform change in such a
way that it only changes the XPaths for fields within repeat groups. But I
might also see if we can instead find a totally different way to solve or
work around this problem. Or, I'll see how I feel about pushing this off
for another year. Fixing choice filters to be able to work will within
repeat groups was not meant to be a huge project.

Thanks again,

Chris

On Wed, Dec 2, 2015 at 2:48 PM Mitch Sundt mitchellsundt@gmail.com wrote:

w.r.t. moving forward...

Even if this particular change impacts a very small number of forms,
we should figure out how to implement non-backward-compatible changes so
that they are safe moving forward. We have no visibility into who uses ODK
Collect and ODK Aggregate and no reliable way to notify them of changes.

This could be a huge deal when or if we ever revise the JR engine to
properly handle timezones, dates and times, and date and time arithmetic.

Or if we find a need to change XLSForm to emit XForms features or
XPath functions that currently don't work (e.g., to use [1] or position(),
or XForms events, for example).

Note that these non-backward-compatible changes extend to not just
XLSForm and ODK Collect, but also to servers. You can use the same revision
tag to indicate new or different processing to be taken on the server.

=============
Managing this change requires:

I believe that this requires:

(1) the XML form definition is the place to define the
feature-enabling-labels that the form requires of the device and server
processing steps.

(2) when the JR engine or any tool implements non-backward-compatible
changes, it only activates them when it detects the presence of those
feature-enabling-label in the XML form definition. Omitting that
feature-enabling-label is a directive to "stay in the past".

(3) ODK Collect and other tools processing the XML form definition or
the XML submission documents should warn the user (e.g., pop up a Toast) if
they are running a version that does not recognize the
feature-enabling-labels in the form's revision tag (i.e., they are given a
form that indicates an advanced feature that they are unaware of) -- this
notification informs the user that they should upgrade that tool to
correctly process the directive. Alternatively, the tool could reject the
form, Force Close or abort, which is the current behavior when ODK
Aggregate or ODK Collect gets an XForms definition that uses a new,
unrecognized, function, etc. ;-<

=============
The stable upgrade path is to:

(1) update your form distribution and data aggregation server(s),
(2) update your collection devices (or collection processing server
-- e.g., Enketo), and
(3) only then update the form generation tool to emit forms with the
non-backward-compatible change.

=============
Specifying the Revision settings:

I believe the XLS file should not contain a setting to select or
specify the revision
.

I believe this should be specified at the time the XLSForm converter
is executed on the source XLS file.
I.e., the web UI for these tools should have a spinner or other UI
element that enables users to choose to "stay in the past"

This forces users to actively stay in the past.

The ODK site will probably achieve this with a drop-down that allows
the user to specify, e.g.,

"full current functionality"
"generate a form definition compatible with ODK Collect 1.4.7 and
earlier"

To allow all groups to independently advance the functionality within
their tool families, the combination of features that each of these
drop-down choices corresponds to should be documented by those groups.

I.e., I see a future where a form generated on Ona for Enketo may not
be compatible with stock ODK Collect unless an appropriate compatibility
drop-down is selected.

This could allow, for example, Survey CTO to implement and recoup
costs for JR date-time fixes or other XForms extensions within their
ecosystem before releasing that fix back to the wider community.

I do not see any value in offering a user the choice of which
features are enabled or disabled. I suppose some sites could offer that,
but it seems overly complex. Users really just need a way to specify
compatibility to whatever older version of the tools they are using.

==============
XLSForm functionality

I am fine with the XLSForm generator taking a comma-separated list of
features to enable, and passing that list through via the revision tag to
the the XML form definition.

Upon reflection, I think this is the only way to manage multiple
independent parallel contributions to the various codebases. It allows
different groups to add and enable different functionalities at different
rates.

We just need to maintain a global list of all of the feature labels
that every group is using so that we don't choose names that collide.

And, some of these feature labels may simply pass through the XLSForm
generator and on to the downstream tools. e.g., date-time fixes might not
have any impact on XLSForm, but would alter how the JR engine processed the
form.

On Wed, Dec 2, 2015 at 10:25 AM, Martijn van de Rijdt < martijn@enketo.org> wrote:

Ok, I was just wondering if it's really worth the effort. KoBo found
3 forms using current(). Not sure how many are actively used still. For
them it would seem to me easier to just contact that/those user(s)
individually to come up with a plan.

I can live with the XForm attribute and XLSForm setting addition.

Chris, to respond to your question about adding support for the bug
to Enketo. I don't currently see a reason to do so, as no Enketo user has
ever relied on the bug and I highly doubt any will in the future. However,
I don't think this should be relevant to this discussion. We're all in
agreement that we want the community to move forward with the correct
syntax.

On Tuesday, December 1, 2015 at 4:58:45 PM UTC-7, Christopher Robert wrote:

We have individual users with 500+ devices in the field. Updating
devices -- or even keeping track of which devices are on which versions --
is a major undertaking. In total, we're talking many thousands of devices
spread across many countries. The problem is not a small one, and our
commitment to backward compatibility is one that our users not only
appreciate but expect.

ODK/SurveyCTO Collect continues to use older versions of forms to
view, edit, or finalize submissions begun under those older versions, even
when newer versions exist. So the backward-compatibility challenge is a
formidable one. If you change the way JavaRosa interprets expressions in
the kind of blanket way that you suggest, then users will be faced with the
impossible choice of running software that fails to function properly with
existing, in-progress, or old submissions, or running software that fails
to function properly with newer submissions. Multiply the difficulty of the
choice and the potential confusion of XPath parsing errors times
however-many devices there are in the field.

No, this is a legitimately difficult situation that requires a
tremendous amount of care. Changing how XPath expressions are interpreted
is a big deal for the Collect world, though I can see why it would be
considerably less so in the Enketo world.

Now, I will admit that there are bound to be very few users who
already use current() in choice filters. Mitch suggested the approach a few
times in the public forum, but even I failed to get it to work the first
several times I tried to follow his advice. So chances are that not many
projects are using current() today. (I know of 1-2, maybe there are 8-10
total. That's just a guess.)

Enketo aside, we could -- today! -- allow people to reference
in-context repeated values by exploiting the current JR support for
current(). For some users/projects, there would be an immediate benefit.
But obviously that relies on "the bug" and so it would expand the potential
need for backward compatibility. And it would break choice-filter
compatibility between Collect and Enketo (say, for SurveyCTO users at
least).

It's easy to say "just fix the bug and then this problem goes
away," but the fact is that we don't quite know how to fix it without
breaking other things; we have a candidate for a fix, but it's seriously
scary. It may take time. And meanwhile, we have already gone multiple years
with users waiting to be able to reference within-context repeated values
in choice filters. If a safe fix drags on for another year or two -- either
because it's just too hard or because nobody quite has the time -- is that
just the way it goes? I don't know. It's not obvious to me that breaking
compatibility with Enketo and relying on current behavior isn't the
welfare-maximizing choice.

In the end, I believe that Mitch's suggestion re: compatibility is
maybe necessary (storing in the XML some indication of how current() in
choice filters should be interpreted). And if we totally screwed up the fix
and broke a bunch of use cases, at least it might offer some escape for
affected users (if we allowed an override on the settings sheet).

So no, I don't agree that it's as simple as "fix the bug and make
people upgrade."

Chris

On Tue, Dec 1, 2015 at 5:11 PM Martijn van de Rijdt < mar...@enketo.org> wrote:

I can't help thinking that we're overthinking this
backwards-compatibility issue.

Ideally we have old clients that have the bug and new clients that
do not have the bug. Isn't that the case with any other major bug fix? If
an old form stops working in the new version of ODK Collect, the survey
owner will either have to upload a new version of the form or the user will
have to downgrade ODK Collect. Surely this will be a problem for some
users, but for how many? A small fraction of one percent seems like a
reasonable guess.

Maybe the backwards-compatibility should be addressed just by
warning users, as Yaw suggested way above.

Any 'side-effects' of fixing the bug, like with creating itemsets,
can be tested for, I presume, before deploying the fix.

One way to detect at least a significant portion of the current
current() users, is to do a regex search for any usage of the
'current()/path/to/node' where current()/ is not followed by '../'. That
usage is specific to this bug to refer to a sibling node and will not occur
any more after the bug is fixed (because a question cannot have children).

On Mon, Nov 30, 2015 at 11:43 AM, Christopher Robert < cro...@surveycto.com> wrote:

I wasn't going to suggest adding the different parameters to any
of the XLSForm templates -- so they wouldn't become cluttered with this
stuff. Really, it would just be an emergency option for somebody requiring
different compatibility for some component that was out of sync with their
other components. Ideally, nobody would actually have to use the settings
sheet to override any of these options.

To be specific, my proposal would be that everybody
support current_always_field=false today, and that this be how pyxform
today defaults. Ideally, this would include Enketo, as Collect and
SurveyCTO's Web Collect (and anything else using the ODK JavaRosa) already
support current_always_field=false as their current behavior. Then, we
figure out how to get current_always_field=true to work in JavaRosa and
phase that in, eventually changing the pyxform default to that once we
think that enough components support it.

We have thousands of devices out there using older Collect
versions, even well after their servers have been updated... so my guess is
that it's some of those users who will end up having to use the
settings-sheet option to force current_always_field=false after we've
changed the default to true.

Does all of that make sense?

Chris

On Sun, Nov 29, 2015 at 6:48 PM Yaw Anokwa yan...@nafundi.com wrote:

All this sounds fine to me, but wanted to raise some potential
issues.
Discrete parameters also means folks can pick and choose what
behavior
they want and it's hard to move the entire community forward
that way.
Also, once you get past two or three, you end up with a settings
sheet
of lots of flags to toggle and that isn't a great user
experience.

Yaw

Need ODK consultants? Nafundi provides form design, server setup,
in-field training, and software development for ODK. Go to
https://nafundi.com to get started.

On Sun, Nov 29, 2015 at 6:47 PM, Christopher Robert cro...@surveycto.com wrote:

Having a number of discrete parameters that are overridable on
the settings
sheet makes sense to me (like "current_references_parent" or
perhaps
"current_always_field"). We could then add model parameters
and change the
pyxform default over time, but users could always override on
their settings
sheets as needed for specific compatibility.

For example, our SurveyCTO pyxform for 2.10 (we've given up on
2.01 now)
could default to current_always_field=false, which would mean
that JR would
be backward-compatible and pyxform would emit relative paths
for the
choice_filter column assuming that current() references the
parent. This
would break all choice_filters in Enketo, but our Enketo users
could set
current_always_field=true in their settings sheets.

For that to work, though, we'd also have to add support for
current_always_field=true in JR. We could do that as Meletis
has proposed,
which may have other side effects -- we'll have to see (e.g.,
for itemset
copying).

Alternatively, we could try to convince Martijn to support
current_always_field=false, which is likely trivial. If he
would do that,
then we could push JR's support for current_always_field=true
to when we
could figure out how to do it safely. Also, it would mean
producing a
version of Enketo that supports relative paths in choice
filters in a manner
compatible with ODK Collect. Ona, Kobo, and other Enketo users
could
rejoice, able to produce forms that use current() (like those
that reference
same-repeat values) in their choice_filters. That boon to
users has to be
worth something, even if a principle is slightly violated
(supporting
something outside the XPath spec).

Chris

On Tue, Nov 24, 2015 at 8:43 PM Alex Dorey dore...@gmail.com wrote:

Hi, Just catching up on the discussion. Sorry if I'm a bit
behind in my
understanding of things.

Regarding the pyxform changes--

  1. Instead of the date, could we have the configuration
    parameter describe
    the specific change?

So instead of:

revision="2015-11-24"

we specify something like:

current_references_parent=True (default to False with the
next pyxform
version bump)

  1. Could we put this configuration into the xlsform's
    settings sheet? I
    think this would be easier to test and configure. We could
    also add a way to
    override the setting with an extra argument to the CLI or
    xls2json call.

On Tuesday, November 24, 2015 at 3:26:18 PM UTC-5, Mitch wrote:

I agree. Having just delved into the itemset code, there are
a number of
issues that would need to be cleaned up.

What I would propose is to leave JR / ODK Collect broken
as-is. This
will be future work.

The python code will then be the focus of the tweaks.

(1) change:

json_survey =

xls2json.workbook_to_json(json.loads(request.POST['workbookJson']),

form_name=form_name, warnings=warningsList)

to add an optional revision argument:

json_survey =

xls2json.workbook_to_json(json.loads(request.POST['workbookJson']),

form_name=form_name, warnings=warningsList,
revision='2015-11-24')

(2) change that implementation to:

(2a) store the revision string somewhere in the returned
json object.

(2b) expand ${fieldname} with the new relative path
expansion only if the
passed-in revision is >= '2015-11-24' ; otherwise, use the
absolute path
expansion.

(3) change:

create_survey_element_from_dict()

to emit the revision value as an attribute on the
element.

===============
The JR itemset bug will be fixed at a later date.

When that is fixed, we can introduce a new revision string
for that
behavior change.

i.e., once we have the revision string moving through the
system, we can
begin to support non-backward-compatible changes to
behaviors in XLSForm and
JR form processing.

For the time being, the ODK XLSForm converter will not
supply a revision
argument, and not make use of this pyxform enhancement. Once
the JR fix is
done, we will then support that new revision and the current
behavior,
defaulting to the new behavior, but supporting users in
generating the old
behavior.

Mitch

On Tue, Nov 24, 2015 at 8:44 AM, Martijn van de Rijdt < mar...@enketo.org> wrote:

I appreciate the creative solution. It does sound like it
will become a
bit of a headache to manage and explain to users though.
Whether it's worth
it depends on how many active forms are out there that rely
on the bug. We
should get Ona's and KoBo's input on this too, I think.

From Enketo's side, relative paths were never supported in
choice-filters (to my embarrassment and surprise - which
led to my
question). As a result, there is no significant issue with
backwards
compatibility in Enketo once it is fixed in JR. Not sure
how meaningful this
is to ODK Collect, but no Enketo user has ever reported
this lack of
support. I think to date it's been too hard to craft such
choice-filters.
Because of the way Enketo is integrated into KoBo, Ona,
Formhub, I'd be
surprised if there were a significant amount of those users
using relative
choice-filters.

I do indeed feel strongly about following the XForms 2.0
spec here. I
think relative path support is going to be very useful in
the future, and I
agree that we need to look at the bigger picture and do
this correctly, so
it won't catch up with us.

On Tue, Nov 24, 2015 at 4:24 AM, Christopher Robert cro...@surveycto.com wrote:

Okay, well, this is turning into a bigger and bigger
project and we may
well miss the window for our 2.01 maintenance release
(and/or make the
changes so extensive and scary that we cannot in good
conscience include
them in a maintenance release).

However, your idea for making JR sensitive to the revision
value
emitted by pyxform opens a new possibility: we could
release the pyxform
change with 2.01 so that users can begin to benefit from
the relative paths
(relying on today's current() behavior), then assume that
when JR is "fixed"
it will only be fixed for XML emitted by newer pyxforms
that would have
updated their paths to account for the fix (using your
revision-value idea).
The primary drawback to this approach is that we would
basically break all
cascading-select filter expressions for SurveyCTO+Enketo
users (since
Martijn clearly wants to implement the "fixed" version of
current() rather
than the JavaRosa-compatible version of current()).

Meletis also found another issue relating to the
evaluation context in
jr:choice-name() (
https://github.com/opendatakit/opendatakit/issues/1180),
so we have that to contend with as well. If we can't
safely resolve the
context for jr:choice-name(), then that would present
another major drawback
and further lessen our enthusiasm for this entire
adventure.

Best,

Chris

On Mon, Nov 23, 2015 at 5:45 PM Mitch Sundt < mitche...@gmail.com> wrote:

And, if it isn't clear, to preserve backward-compatible
operations,
XLSForm should not expand ${fieldname} with a relative
path unless the
revision value is the newer one; if the revision is not
specified, it would
emit absolute paths like it currently does.

I also switched to use a date value for the revision value
("2015-11-23") which allows simple lexical comparisons.

On Mon, Nov 23, 2015 at 1:03 PM, Mitch Sundt < mitche...@gmail.com> wrote:

w.r.t. backward compatibility

I too am concerned.

A proposed solution:

  1. Add a command line argument or other mechanism to
    request that
    XLSForm inject a 'revision="2015-11-23"' attribute into
    the element.
    e.g.,

This is used to alter the behavior of the XForms engine
to
accommodate backward-compatible bug behaviors.

We can't identify the bugs (e.g,
'legacy="itemset-parent"'), since
that would break older forms run on the new ODK Collect
releases.
We have to advance the revision code as bugs are fixed.

I think an integer value is fine for this.

I don't anticipate making a lot of revisions, and I
think the
revisions would be sequentially applied (you can't
pick-and-choose what bugs
you include).

In the future, this would allow us to re-do the
date-time behavior.

  1. Modify the XLSForm tools (e.g., the online form
    generator) to
    enable the user to select what revision they want to
    emit (much like we did
    last year with the JR form eval choice setting within
    ODK Collect). We can
    default to emitting the latest revision if the user does
    not make a choice.

  2. Augment JR to look for this revision attribute and
    store/restore
    it in the FormDef binary object.

Based upon this value, alter the behavior of the JR
itemset
evaluation to fix the path bug.

  1. Modify ODK Validate to scan for syntax that may be
    impacted by a
    revision (to advise whether to change something). Not
    sure how complex that
    might be. May not be always possible?

w.r.t. impacts of change

Thank you Meletis for researching when this change
occurred.

I will have to ponder what effect the parent/node change
would have
on itemset copying. Hadn't thought that there was any
relationship between
the two, but perhaps there is.

Yaw also reported that the Cascading Select itemsets on
opendatakit.appspot.com are not working with the 1.4.7
rev 1053 release (and
they work but give odd 'null' resolutions on 1057). This
is in a related
area of this same code, so I will hopefully gain a
better understanding of
dynamic itemsets as I uncover the causes of that problem.


Mitch

On Mon, Nov 23, 2015 at 5:59 AM, Christopher Robert cro...@surveycto.com wrote:

So just to be clear, the "fix" we're testing here
reverts the entire
context within that populateDynamicChoices() function,
to be the field
instead of the parent -- so the effects are potentially
much wider than just
what current() returns. Potentially anything and
everything that depends on
context can change now, since the context will shift
from parent to field.

This means that we revert a change from 5 years ago and
alter the
context from now on. Neither Meletis nor I feel
terribly confident that we
can entirely predict the consequences to backward
compatibility. We're
testing the things we can think to test, but honestly
it's XML users in the
broader ODK community who face the greatest risks here;
SurveyCTO users by
and large use XLSForm and thus the potential XPath
syntax consequences are
sharply limited.

Are you guys very sure that you want to take this path?
I can tell
you now that if we make this change and then there is
some outcry after the
next ODK Collect release, SurveyCTO Collect will very
likely stick with the
new change (since we will have changed pyxform and
we'll then have too many
backward-compatibility concerns to then reverse the
change).

Thanks,

Chris

On Mon, Nov 23, 2015 at 5:23 AM mel...@surveycto.com wrote:

Just for the history's sake, I found the time where
the behavior
changed. It was at March 11, 2010:

https://bitbucket.org/m.sundt/javarosa/commits/b41875b2c23c9a21a868a5d2dc0ffc1f6b5b0b65#Lcore/src/org/javarosa/core/model/FormDef.javaT635

Take a look only in the changes of the
"populateDynamicChoices"
method. The context reference used to be the "current"
field, but after the
changes above it started be the field's parent (by
adding getParent()).
Perhaps there was a reason for that change, but
perhaps not, but I think all
of us agree that this should be fixed, even after 5
and a half years.

Meletis

On Monday, November 23, 2015 at 12:08:47 PM UTC+2, mel...@surveycto.com wrote:

Hi Mitch,

Please take a look at the code comment in line 31
here:

https://bitbucket.org/m.sundt/javarosa/src/bf17fb749b74e53915f086ca3827c3c6fe73fd07/core/src/org/javarosa/core/model/ItemsetBinding.java?at=default&fileviewer=file-view-default#ItemsetBinding.java-31

It clearly says: "ref of the control parent
(group/formdef) of
itemset question"

So, for some reason, they wanted it to point to the
parent. I also
see that when the dynamic choices are populated, the
evaluation context is
wrapped around that parent (contextRef object),
that's why current() refers
to the parent group when evaluating itemsets nodesets.

I will run a few tests by changing the contextRef
field value, I
just wanted to mention that in case you remember a
reason for that code
comment.

Meletis

On Friday, November 20, 2015 at 8:49:12 PM UTC+2, Mitch wrote:

Yes, I think it should be a one-line change; just
don't know
where.

Mitch

(I do jar-updates over Christmas / New-Years.
Tedious, but
doesn't require a whole lot of attention.)

On Fri, Nov 20, 2015 at 10:36 AM, < mel...@surveycto.com> wrote:

Hi Mitch,

Thank you for all the details, I guess I was
missing some of the
context of the original issue.

We will give it a shot to see if we can come up
with a few ideas
how to fix the evaluation content of the itemset
nodeset. I just hope it
will not be like last year with the "form
processing logic" in JR, otherwise
I believe that every Christmas will be a JR
Christmas. :slight_smile:

Meletis

On Friday, November 20, 2015 at 7:29:14 PM UTC+2, Mitch wrote:

The form was attached in one of my earlier replies.

On Fri, Nov 20, 2015 at 9:27 AM, Mitch Sundt mitche...@gmail.com wrote:

And you see this in constraint expressions where
you write:

. > 6

I.e., current()/. should refer to the field, and
not the group
containing the field, as it currently does when
evaluating expressions in
the itemset nodeset.

On Fri, Nov 20, 2015 at 9:26 AM, Mitch Sundt mitche...@gmail.com wrote:

The issue is that there is a discrepancy in the
meaning of
'.' across the different contexts of: calculate,
relevant, , and
these itemset nodeset expressions.

In other usages, you need to reference this
other field with
current()/../filterfield ; but in the nodeset
expression, you would
reference it with current()/./filterfield

I.e., the notion of '.' is different when used
within a
nodeset expression and when used within a
constraint expression or an
field.

See the attached NBiggestOfSet form. For the
label, it has:

    <select1

ref="/NBiggestOfSet/education/edu_level">

      <label>What is <output

value="current()/../referred_attendee_name"/>'s
level of education?
...

For a constraint, it has:
<bind

calculate="/NBiggestOfSet/attendee/name[position(..)=int(current()/../referred_attendee)]"

nodeset="/NBiggestOfSet/education/referred_attendee_name" readonly="true()"

type="string"/>

These each use the current()/../ to reference
the group (
/NBiggesetOfSet/education ).

But, within the nodeset definition (if this form
used one),
you would need to use
current()/./referred_attendee_name to reference a
field in that group.

On Fri, Nov 20, 2015 at 8:22 AM, < mel...@surveycto.com> wrote:

Hi Mitch,

I think I'm missing something but I think that
nothing is
broken and that the only issue is that the
expression
current()/../filterfield in your example above
should be just changed to
current()/filterfield. If you make that change,
everything should work
because it will be resolved to:

/data/group1/filterfield

Why should a "../" be present there? We have
used
current()/somefield successfully in one of our
test forms and it works fine.
And I think it would also work great in your
example. Wouldn't it? Or am I
totally out of context?

Meletis

On Friday, November 20, 2015 at 5:35:16 PM UTC+2, Mitch wrote:

Hi Meletis,

That's not the issue --

The issue is that if you use a filter
condition in the
&g

--
Revolutionizing data collection since 2012.

Enketo https://enketo.org/ | LinkedIn
http://www.linkedin.com/company/enketo-llc | GitHub
https://github.com/enketo | Twitter
https://twitter.com/enketo | Blog http://blog.enketo.org/

--
You received this message because you are subscribed to the Google
Groups "ODK Developers" group.
To unsubscribe from this group and stop receiving emails from it,
send an email to opendatakit-developers+unsubscribe@googlegroups.com
.
For more options, visit https://groups.google.com/d/optout.

--
Mitch Sundt
Software Engineer
University of Washington
mitchellsundt@gmail.com

--
You received this message because you are subscribed to the Google
Groups "ODK Developers" group.
To unsubscribe from this group and stop receiving emails from it,
send an email to opendatakit-developers+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google
Groups "ODK Developers" group.
To unsubscribe from this group and stop receiving emails from it, send
an email to opendatakit-developers+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--
Mitch Sundt
Software Engineer
University of Washington
mitchellsundt@gmail.com

--
You received this message because you are subscribed to the Google
Groups "ODK Developers" group.
To unsubscribe from this group and stop receiving emails from it, send
an email to opendatakit-developers+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to a topic in the
Google Groups "ODK Developers" group.
To unsubscribe from this topic, visit
https://groups.google.com/d/topic/opendatakit-developers/nN8HTA5pUIM/unsubscribe
.
To unsubscribe from this group and all its topics, send an email to
opendatakit-developers+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--

Revolutionizing data collection since 2012.

Enketo https://enketo.org/ | LinkedIn
http://www.linkedin.com/company/enketo-llc | GitHub
https://github.com/enketo | Twitter https://twitter.com/enketo
| Blog http://blog.enketo.org/