ODK Collect vs ODK Aggregate 'boolean' type support

Xiphware · April 17, 2017, 9:01am

I've been playing around using a custom XForms client with ODK Aggregate, and noticed some behavior around the support (or lack of?) of the 'boolean' type for property bindings. That is, something like

... ...

So I had my custom XForms client submit a variety of different values in the uploaded XML instance to try to determine what exactly ODK Aggregate expects to see for one of these 'boolean' properties. The results were:

'true' -> true
'false' -> false
'T' -> true
'F' -> false
'Y' -> true
'N' -> false
'yes' -> true
'no' -> false
'0' - > false
'1' -> false
and just for fun
'oui' -> false
'non' -> false

So it appears that ODK Aggregate does indeed recognize the 'boolean' type, in as much as it is clearly processing the corresponding data element as a boolean literal and inserting true or false in the database accordingly. Oddly, however, it does not seem to recognize '0' and '1' as legitimate boolean literals.

Note, if you subsequently fetch this submission from ODK Aggregate, you will always get 'true' or 'false' back in the XML instance. So, arguably, you really have to use 'true' and 'false' everywhere in your XForm when dealing with boolean elements, specifically when checking property values in XPath expressions (eg relevant="/data/yesorno = 'true'").

So my first question is can someone on the ODK Aggregate dev team confirm that yes in fact ODK Aggregate does support the 'boolean' data type, as I've described above?

I ask because it would appear that the client ODK Collect does not support a boolean type (!). Specifically, when presented with an XForm with a boolean element, the corresponding control is just presented as regular string input, into which you can type (and ODK Collect will accept) whatever text you type in (which will get shoved verbatim into the XML sent up to ODK Aggregate. So if, for example, you typed in "1" you'd actually be submitting false). Further, if you try to use the boolean data type in XLSForm the translation parser will throw an "unrecognized type" error. Which supports my conclusion that ODK COllect doesnt actually support booleans; rather it simply ignores the type and treats the binding as the default string type instead.

In summary, ODK Aggregate does appear to support boolean properties, but ODK Collect (and XLSForm) does not. Correct? [that seems very odd...]

Gareth

Mitch_S · April 17, 2017, 5:55pm

Boolean is available in all 3.

The "acknowledge" prompt type in XLSForm / ODK Collect is interpreted as
Boolean in ODK Aggregate.

ODK Aggregate uses the following parsing logic to determine whether
something is true:

github.com

getodk/aggregate/blob/master/src/main/java/org/opendatakit/common/utils/WebUtils.java#L151


      
          /*
           * Copyright (C) 2011 University of Washington
           * Copyright (C) 2018 Nafundi
           *
           * Licensed under the Apache License, Version 2.0 (the "License"); you may not
           * use this file except in compliance with the License. You may obtain a copy of
           * the License at
           *
           * http://www.apache.org/licenses/LICENSE-2.0
           *
           * Unless required by applicable law or agreed to in writing, software
           * distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
           * WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
           * License for the specific language governing permissions and limitations under
           * the License.
           */
          
          package org.opendatakit.common.utils;
          
          import java.io.BufferedReader;
          import java.io.IOException;
          import java.io.InputStream;
          import java.io.InputStreamReader;
          import java.io.UnsupportedEncodingException;
          import org.apache.http.HttpEntity;
          import org.apache.http.HttpResponse;
          import org.opendatakit.common.web.constants.BasicConsts;
          import org.opendatakit.common.web.constants.HtmlConsts;
          import org.slf4j.Logger;
          import org.slf4j.LoggerFactory;
          
          public class WebUtils {
          
            static final String IS_FORWARD_CURSOR_VALUE_TAG = "isForwardCursor";
            static final String URI_LAST_RETURNED_VALUE_TAG = "uriLastReturnedValue";
            static final String ATTRIBUTE_VALUE_TAG = "attributeValue";
            static final String ATTRIBUTE_NAME_TAG = "attributeName";
            static final String CURSOR_TAG = "cursor";
            static final Logger logger = LoggerFactory.getLogger(WebUtils.class);
          
            private WebUtils() {
            }
          
            public static Boolean parseBoolean(String value) {
              Boolean b = null;
              if (value != null && value.length() != 0) {
                b = Boolean.FALSE;
                if (value.compareToIgnoreCase("ok") == 0) {
                  b = Boolean.TRUE;
                } else if (value.compareToIgnoreCase("yes") == 0) {
                  b = Boolean.TRUE;
                } else if (value.compareToIgnoreCase("true") == 0) {
                  b = Boolean.TRUE;
                } else if (value.compareToIgnoreCase("T") == 0) {
                  b = Boolean.TRUE;
                } else if (value.compareToIgnoreCase("Y") == 0) {
                  b = Boolean.TRUE;
                }
              }
              return b;
            }
          
            public static String readResponse(HttpResponse resp) throws IOException {
          
              HttpEntity e = resp.getEntity();
              if (e != null) {
                return WebUtils.readResponseHelper(e.getContent());
              }
          
              return BasicConsts.EMPTY_STRING;
            }
          
            public static String readGoogleResponse(com.google.api.client.http.HttpResponse resp) throws IOException {
              if (resp != null) {
                return WebUtils.readResponseHelper(resp.getContent());
              }
          
              return BasicConsts.EMPTY_STRING;
            }
          
            private static String readResponseHelper(InputStream content) {
              StringBuffer response = new StringBuffer();
          
              if (content != null) {
                // TODO: this section of code is possibly causing 'WARNING: Going to
                // buffer
                // response body of large or unknown size. Using getResponseBodyAsStream
                // instead is recommended.'
                // The WARNING is most likely only happening when running appengine
                // locally,
                // but we should investigate to make sure
                BufferedReader reader = null;
                InputStreamReader isr = null;
                try {
                  reader = new BufferedReader(isr = new InputStreamReader(content, HtmlConsts.UTF8_ENCODE));
                  String responseLine;
                  while ((responseLine = reader.readLine()) != null) {
                    response.append(responseLine);
                  }
                } catch (UnsupportedEncodingException ex) {
                  ex.printStackTrace();
                } catch (IllegalStateException ex) {
                  ex.printStackTrace();
                } catch (IOException ex) {
                  ex.printStackTrace();
                } finally {
                  try {
                    if (reader != null) {
                      reader.close();
                    }
                  } catch (IOException ex) {
                    // no-op
                  }
                  try {
                    if (isr != null) {
                      isr.close();
                    }
                  } catch (IOException ex) {
                    // no-op
                  }
                }
              }
              return response.toString();
            }
          
          }

···

On Mon, Apr 17, 2017 at 2:01 AM, wrote:

I've been playing around using a custom XForms client with ODK Aggregate,
and noticed some behavior around the support (or lack of?) of the 'boolean'
type for property bindings. That is, something like
... ...
So I had my custom XForms client submit a variety of different values in
the uploaded XML instance to try to determine what exactly ODK Aggregate
expects to see for one of these 'boolean' properties. The results were:

'true' -> true
'false' -> false
'T' -> true
'F' -> false
'Y' -> true
'N' -> false
'yes' -> true
'no' -> false
'0' - > false
'1' -> false
and just for fun
'oui' -> false
'non' -> false

So it appears that ODK Aggregate does indeed recognize the 'boolean' type,
in as much as it is clearly processing the corresponding data element as a
boolean literal and inserting true or false in the database accordingly.
Oddly, however, it does not seem to recognize '0' and '1' as legitimate
boolean literals.

Note, if you subsequently fetch this submission from ODK Aggregate, you
will always get 'true' or 'false' back in the XML instance. So, arguably,
you really have to use 'true' and 'false' everywhere in your XForm when
dealing with boolean elements, specifically when checking property values
in XPath expressions (eg relevant="/data/yesorno = 'true'").

So my first question is can someone on the ODK Aggregate dev team confirm
that yes in fact ODK Aggregate does support the 'boolean' data type, as
I've described above?

I ask because it would appear that the client ODK Collect does not
support a boolean type (!). Specifically, when presented with an XForm with
a boolean element, the corresponding control is just presented as regular
string input, into which you can type (and ODK Collect will accept)
whatever text you type in (which will get shoved verbatim into the XML sent
up to ODK Aggregate. So if, for example, you typed in "1" you'd actually be
submitting false). Further, if you try to use the boolean data type in
XLSForm the translation parser will throw an "unrecognized type" error.
Which supports my conclusion that ODK COllect doesnt actually support
booleans; rather it simply ignores the type and treats the binding as the
default string type instead.

In summary, ODK Aggregate does appear to support boolean properties, but
ODK Collect (and XLSForm) does not. Correct? [that seems very odd...]

Gareth

--
You received this message because you are subscribed to the Google Groups
"ODK Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to opendatakit-developers+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--
Mitch Sundt
Software Engineer
University of Washington
mitchellsundt@gmail.com

yanokwa · April 17, 2017, 9:55pm

Hi Gareth,

This is a very good question. Mitch is correct, but the details are
useful to understand, so I wanted to share what I've found.

JavaRosa, the ODK XForms library we use, does have support for
booleans and the relevant code is at
https://github.com/opendatakit/javarosa/blob/master/src/org/javarosa/xpath/expr/XPathFuncExpr.java#L175.
So you can do things like boolean(path/to/node) and
boolean-from-string(/path/to/node) and it will work great.

And while there is a BooleanData type in JavaRosa at
https://github.com/opendatakit/javarosa/blob/master/src/org/javarosa/core/model/data/BooleanData.java,
Collect doesn't have a widget that saves answers as a boolean. So if
you set boolean as the datatype in the XML, it'll default to string.
You can confirm this behavior at
https://github.com/opendatakit/collect/blob/master/collect_app/src/main/java/org/odk/collect/android/widgets/WidgetFactory.java#L51

The fact that acknowledge saves either "OK" or null and has a trigger
control type doesn't seem like the right way to do this. I committed
that code eight years ago at

and I can't remember why I thought it was a good idea. It might not
have been.

Hélène has filed an issue at to fix it at
https://github.com/opendatakit/collect/issues/908 so you can track
that issue for updates.

We want to track down and fix all these problems and your help would
be greatly appreciated. You can find the folks working on the spec at
http://slack.opendatakit.org in the #xforms-spec channel. You can also
find and file issues we know about at
https://github.com/opendatakit/xforms-spec.

Thanks,

Yaw

···

On Mon, Apr 17, 2017 at 10:55 AM, Mitch Sundt wrote: > Boolean is available in all 3. > > The "acknowledge" prompt type in XLSForm / ODK Collect is interpreted as > Boolean in ODK Aggregate. > > ODK Aggregate uses the following parsing logic to determine whether > something is true: > https://github.com/opendatakit/aggregate/blob/master/src/main/java/org/opendatakit/common/utils/WebUtils.java#L151 > > > > On Mon, Apr 17, 2017 at 2:01 AM, wrote: >> >> I've been playing around using a custom XForms client with ODK Aggregate, >> and noticed some behavior around the support (or lack of?) of the 'boolean' >> type for property bindings. That is, something like >> >> >> >> >> ... >> >> ... >> >> >> So I had my custom XForms client submit a variety of different values in >> the uploaded XML instance to try to determine what exactly ODK Aggregate >> expects to see for one of these 'boolean' properties. The results were: >> >> 'true' -> true >> 'false' -> false >> 'T' -> true >> 'F' -> false >> 'Y' -> true >> 'N' -> false >> 'yes' -> true >> 'no' -> false >> '0' - > false >> '1' -> false >> and just for fun >> 'oui' -> false >> 'non' -> false >> >> So it appears that ODK Aggregate does indeed recognize the 'boolean' type, >> in as much as it is clearly processing the corresponding data element as a >> boolean literal and inserting true or false in the database accordingly. >> Oddly, however, it does not seem to recognize '0' and '1' as legitimate >> boolean literals. >> >> Note, if you subsequently fetch this submission from ODK Aggregate, you >> will always get 'true' or 'false' back in the XML instance. So, arguably, >> you really have to use 'true' and 'false' everywhere in your XForm when >> dealing with boolean elements, specifically when checking property values in >> XPath expressions (eg relevant="/data/yesorno = 'true'"). >> >> So my first question is can someone on the ODK Aggregate dev team confirm >> that yes in fact ODK Aggregate *does* support the 'boolean' data type, as >> I've described above? >> >> I ask because it would appear that the client ODK Collect does *not* >> support a boolean type (!). Specifically, when presented with an XForm with >> a boolean element, the corresponding control is just presented as regular >> string input, into which you can type (and ODK Collect will accept) whatever >> text you type in (which will get shoved verbatim into the XML sent up to ODK >> Aggregate. So if, for example, you typed in "1" you'd actually be submitting >> false). Further, if you try to use the boolean data type in XLSForm the >> translation parser will throw an "unrecognized type" error. Which supports >> my conclusion that ODK COllect doesnt actually support booleans; rather it >> simply ignores the type and treats the binding as the default string type >> instead. >> >> >> In summary, ODK Aggregate does appear to support boolean properties, but >> ODK Collect (and XLSForm) does not. Correct? [that seems very odd...] >> >> - Gareth >> >> -- >> You received this message because you are subscribed to the Google Groups >> "ODK Developers" group. >> To unsubscribe from this group and stop receiving emails from it, send an >> email to opendatakit-developers+unsubscribe@googlegroups.com. >> For more options, visit https://groups.google.com/d/optout. > > > > > -- > Mitch Sundt > Software Engineer > University of Washington > mitchellsundt@gmail.com > > -- > You received this message because you are subscribed to the Google Groups > "ODK Developers" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to opendatakit-developers+unsubscribe@googlegroups.com. > For more options, visit https://groups.google.com/d/optout.

Xiphware · April 19, 2017, 12:48am

Thanks for both your replies.

WebUtils.java is certainly a goldmine of important information, and pretty much tells me everything I need to know regarding what Aggregate expects to see for booleans (case-insensitive, yadda, yadda...). I also see that it describes how date strings need to be represented in submissions! which is something I already had to reverse-engineer and figure out somewhat by trial and error... Anyway, great stuff! Is this documented anywhere, or really just in the source code? Its inconsequential if yer staying within the ODK Collect-Aggregate ecosystem, but these formats & constants become pretty important when trying to write other tools to interface with them, especially for submission.

And yes, I dug around the Aggregate source too and I rather have to agree that whereas ODK Aggregate does properly support booleans, pretty much by virtue of being build on JavaRosa, arguably ODK Collect really does not. Instead it gives you a trigger with which to set a field to 'true', and then pushes this up via a string instead. Sure, does the job, but I do rather think using a trigger just to set a boolean value - which really is not that different than, say, an integer field - is a bit of an awkward way to go about it... But then hindsight's 20-20, eh It would be nice if ODK Collect did implement a boolean element type the 'right' way, if only because then XLSForm will generate a corresponding simple XML file for basic forms, without having to introduce triggers!

And thank you for the invitation to participate in working on your spec. I actually come from a standard background - I helped write a lot of the DMTF standards on virtualization in my previous life [perhaps that will explain why I may be appearing a little pedantic... ;-)] I will certainly try to find the time to particulate.

Cheers,

Gareth

Mitch_S · April 20, 2017, 4:45pm

Most everything is just in source code. The developer documentation, such
as it is, is on the github site
https://github.com/opendatakit/opendatakit/wiki

ODK Aggregate was written to conform to the behavior of ODK Collect.

The broader datetime parsing supported by ODK Aggregate is to enable users
to enter dates in row filters on the website in a more lenient and natural
way. For XML submissions containing dates, the roughly-ISO 8601
representation used in ODK Collect should be followed. That fact that ODK
Aggregate allows alternate representations in that field is a non-standard
behavior.

All data formatting and handling in the ecosystem starts with javarosa and
the set of widgets ODK Collect supports.

···

On Tue, Apr 18, 2017 at 5:48 PM, wrote:

Thanks for both your replies.

WebUtils.java is certainly a goldmine of important information, and pretty
much tells me everything I need to know regarding what Aggregate expects to
see for booleans (case-insensitive, yadda, yadda...). I also see that it
describes how date strings need to be represented in submissions! which is
something I already had to reverse-engineer and figure out somewhat by
trial and error... Anyway, great stuff! Is this documented anywhere, or
really just in the source code? Its inconsequential if yer staying within
the ODK Collect-Aggregate ecosystem, but these formats & constants become
pretty important when trying to write other tools to interface with them,
especially for submission.

And yes, I dug around the Aggregate source too and I rather have to agree
that whereas ODK Aggregate does properly support booleans, pretty much by
virtue of being build on JavaRosa, arguably ODK Collect really does not.
Instead it gives you a trigger with which to set a field to 'true', and
then pushes this up via a string instead. Sure, does the job, but I do
rather think using a trigger just to set a boolean value - which really is
not that different than, say, an integer field - is a bit of an awkward way
to go about it... But then hindsight's 20-20, eh It would be nice if
ODK Collect did implement a boolean element type the 'right' way, if only
because then XLSForm will generate a corresponding simple XML file for
basic forms, without having to introduce triggers!

And thank you for the invitation to participate in working on your spec. I
actually come from a standard background - I helped write a lot of the DMTF
standards on virtualization in my previous life [perhaps that will explain
why I may be appearing a little pedantic... ;-)] I will certainly try to
find the time to particulate.

Cheers,

Gareth

--
You received this message because you are subscribed to the Google Groups
"ODK Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to opendatakit-developers+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--
Mitch Sundt
Software Engineer
University of Washington
mitchellsundt@gmail.com