ODK Collect: Updating other questions' variables, dynamic relevant/constraints, looping, etc

[Joseph_E_Flack_IV]

Hello everyone,

My organization appears to be running into some dead ends when it comes to
adding newer, more complex question flows for some upcoming 'modules' that
we would like to include in some of our questionnaires.

The particular pattern we're looking for is to account for the history of a
particular person over a period of 5 years, going chronologically
backwards. But we also want to be able to flip through the responses that
were given and change them if need be.

This would be a repeat of groups. Each group would be 3 questions: start
date of period, end date of period, and status during that period.

Each question group would represent a period. We would want that during
each repeat, the end date of the current period should not occur after the
start date of a previously answered period. This seems possible with
currently. However if one were to go back and change a previous response,
we would want the answer to not conflict with a subsequent response.

For the constraint, it would be nice if we could loop through all the
responses for that repeat and make sure nothing is in conflict. Or at
least, a warning note after the repeat to loop through and see if anything
is in conflict.

Is there any way to do this in ODK Collect? If not, if someone could please
point me to good locations in the source code to make edits, that would be
wonderful.

[Joseph_E_Flack_IV]

I never realized that ODK 2.0 even existed. Maybe this is something that we can consider, but it seems like adopting it is quite a rigorous undertaking. Is it possible to achieve what we need with 1.0?

[Joseph_E_Flack_IV]

I haven't yet went out to create a form in ODK 2.0's 'Survey', but I am not
yet seeing a solution to this issue, though I could be missing something.

I decided to post over in the JavaRosa group as well, with much more detail
as to the specific kind of syntax / functionality we are looking
for: https://groups.google.com/forum/#!topic/javarosa-developers/6Z1fSlJdb7Q

[Mitch_S]

The key limitation of ODK 1.x is that it does not easily support updating
of existing, already-collected, data.

There are many work-arounds for this, but this is fundamental in its
design. Others have written different servers that do support accepting
revisions to existing submissions, and mechanisms for making this work with
the ODK 1.x tools. Look at the compatible server offerings.

The ODK 2.0 tools are intended and designed from the beginning to support
this use case.

···

On Sun, Nov 6, 2016 at 10:01 PM, Joseph E. Flack IV wrote:

I haven't yet went out to create a form in ODK 2.0's 'Survey', but I am
not yet seeing a solution to this issue, though I could be missing
something.

I decided to post over in the JavaRosa group as well, with much more
detail as to the specific kind of syntax / functionality we are looking
for: https://groups.google.com/forum/#!topic/javarosa-
developers/6Z1fSlJdb7Q

--
You received this message because you are subscribed to the Google Groups
"ODK Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to opendatakit-developers+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--
Mitch Sundt
Software Engineer
University of Washington
mitchellsundt@gmail.com

[Joseph_E_Flack_IV]

Hello Mitch,

I'm really happy to see your reply! Thanks for taking the time to address
my issue.

We are actually quite keen at the idea of implementing ODK 2.0, for various
reasons. However I am wondering if this does indeed address the use case
that we are currently experiencing. Referencing existing submissions is
something that we did indeed have to reach outside of the ODK 1.0 box to
fulfill. We developed an internal solution some time ago.

However the current use case that we are having problems with right now is
a little different than what you mention in two respects.

Firstly, we would like to loop through data. As an arbitrary, easy example,
lets say we wanted to declare a simple loop that would check through all
question responses of a previously submitted form, and see if any of the
answers had the value "foo". We want to use the same, simple syntax,
whether there were a few questions in the form, or hundreds.

The second thing we need is a way to reference / do this with question
responses in the currently active, un-submitted, and unsaved form.

Are either of these things possible with ODK 2.0? If not, this seems like a
limitation with ODK Xform Specification
https://opendatakit.github.io/odk-xform-spec/ and syntax (previously
JavaRosa / OpenRosa). I love the declarative style of the spec. I searched
for any turing complete declarative languages (which should have loops),
and found XLST (XLST Language https://en.wikipedia.org/wiki/XSLT, XLST
Elements https://en.wikipedia.org/wiki/XSLT_elements).

If anyone happens to know any way to accomplish a loop as described using
ODK, I will most certainly implement it. Otherwise it looks like we may
have to use some kind of workaround.

···

>

[Mitch_S]

The big change with the ODK 2.0 tools is a switch to Javascript, abandoning
XForms and the idiosyncrasies of the Javarosa library implementation of
that standard.

The ODK Survey tool uses a spreadsheet to define your survey, which is then
compiled into a Javascript definition that is interpreted on the device.
The spreadsheet syntax is similar but different from ODK 1.x. In that 2.0
syntax, we have abandoned repeat groups and relevance and untangled
data-validation from the order in which you traverse a form. Instead, as a
form developer, you have: if-then-else, goto with branch labels, and a "do
section" command that works somewhat like a subroutine; and at any point in
the form, you can validate subsets of field values (i.e., is everything OK
so far?). The base structure is the data table in which the data for a form
is stored -- the form works on top of that data table. Repeat groups are
handled through linkages to data stored in other data tables (and there is
no strict parent-child relationship -- it can be whatever linkage makes
sense for your application). Finally, you can have any number of forms for
the same data table, each potentially modifying only a subset of fields in
that table.

I believe this would allow you to stay within a given survey and just keep
looping through it. You would end up with only one final record of field
values whenever you finalized the form. If you wanted to create an ongoing
record of all of your changes, you would need to create new data rows for
each field value change. That could be done in ODK Tables, but you would
lose the easy spreadsheet-based description of your survey (we worked with
the Jane Goodall Institute on a chimpanzee activity survey that directly
recorded interaction events in this way).

ODK Tables provides a means to display raw html and Javascript and is
intended for displaying and selecting or navigating through
already-captured data to either launch ODK Survey, make immediate updates
to that data, or simply for visualizing those results (e.g., using a
Javascript graphing library like d3).

···

On Mon, Nov 7, 2016 at 7:39 PM, Joseph E. Flack IV wrote:

Hello Mitch,

I'm really happy to see your reply! Thanks for taking the time to address
my issue.

We are actually quite keen at the idea of implementing ODK 2.0, for
various reasons. However I am wondering if this does indeed address the use
case that we are currently experiencing. Referencing existing submissions
is something that we did indeed have to reach outside of the ODK 1.0 box to
fulfill. We developed an internal solution some time ago.

However the current use case that we are having problems with right now is
a little different than what you mention in two respects.

Firstly, we would like to loop through data. As an arbitrary, easy
example, lets say we wanted to declare a simple loop that would check
through all question responses of a previously submitted form, and see if
any of the answers had the value "foo". We want to use the same, simple
syntax, whether there were a few questions in the form, or hundreds.

The second thing we need is a way to reference / do this with question
responses in the currently active, un-submitted, and unsaved form.

Are either of these things possible with ODK 2.0? If not, this seems like
a limitation with ODK Xform Specification
https://opendatakit.github.io/odk-xform-spec/ and syntax (previously
JavaRosa / OpenRosa). I love the declarative style of the spec. I searched
for any turing complete declarative languages (which should have loops),
and found XLST (XLST Language https://en.wikipedia.org/wiki/XSLT, XLST
Elements https://en.wikipedia.org/wiki/XSLT_elements).

If anyone happens to know any way to accomplish a loop as described using
ODK, I will most certainly implement it. Otherwise it looks like we may
have to use some kind of workaround.

--
You received this message because you are subscribed to the Google Groups
"ODK Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to opendatakit-developers+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--
Mitch Sundt
Software Engineer
University of Washington
mitchellsundt@gmail.com

[Mitch_S]

And it is possible to extend the javascript within ODK Survey (by adding
your own prompt type) to save off a copy of your collected data at a given
point within each loop of your survey. That would not requirewriting your
own Javascript to extend the Javascript that handles all of the ODK Survey
navigation but would not require any modification to the Java software.

···

On Tue, Nov 8, 2016 at 10:12 AM, Mitch Sundt wrote:

The big change with the ODK 2.0 tools is a switch to Javascript,
abandoning XForms and the idiosyncrasies of the Javarosa library
implementation of that standard.

The ODK Survey tool uses a spreadsheet to define your survey, which is
then compiled into a Javascript definition that is interpreted on the
device. The spreadsheet syntax is similar but different from ODK 1.x. In
that 2.0 syntax, we have abandoned repeat groups and relevance and
untangled data-validation from the order in which you traverse a form.
Instead, as a form developer, you have: if-then-else, goto with branch
labels, and a "do section" command that works somewhat like a subroutine;
and at any point in the form, you can validate subsets of field values
(i.e., is everything OK so far?). The base structure is the data table in
which the data for a form is stored -- the form works on top of that data
table. Repeat groups are handled through linkages to data stored in other
data tables (and there is no strict parent-child relationship -- it can be
whatever linkage makes sense for your application). Finally, you can have
any number of forms for the same data table, each potentially modifying
only a subset of fields in that table.

I believe this would allow you to stay within a given survey and just keep
looping through it. You would end up with only one final record of field
values whenever you finalized the form. If you wanted to create an ongoing
record of all of your changes, you would need to create new data rows for
each field value change. That could be done in ODK Tables, but you would
lose the easy spreadsheet-based description of your survey (we worked with
the Jane Goodall Institute on a chimpanzee activity survey that directly
recorded interaction events in this way).

ODK Tables provides a means to display raw html and Javascript and is
intended for displaying and selecting or navigating through
already-captured data to either launch ODK Survey, make immediate updates
to that data, or simply for visualizing those results (e.g., using a
Javascript graphing library like d3).

On Mon, Nov 7, 2016 at 7:39 PM, Joseph E. Flack IV joeflack4@gmail.com wrote:

Hello Mitch,

I'm really happy to see your reply! Thanks for taking the time to address
my issue.

We are actually quite keen at the idea of implementing ODK 2.0, for
various reasons. However I am wondering if this does indeed address the use
case that we are currently experiencing. Referencing existing submissions
is something that we did indeed have to reach outside of the ODK 1.0 box to
fulfill. We developed an internal solution some time ago.

However the current use case that we are having problems with right now
is a little different than what you mention in two respects.

Firstly, we would like to loop through data. As an arbitrary, easy
example, lets say we wanted to declare a simple loop that would check
through all question responses of a previously submitted form, and see if
any of the answers had the value "foo". We want to use the same, simple
syntax, whether there were a few questions in the form, or hundreds.

The second thing we need is a way to reference / do this with question
responses in the currently active, un-submitted, and unsaved form.

Are either of these things possible with ODK 2.0? If not, this seems like
a limitation with ODK Xform Specification
https://opendatakit.github.io/odk-xform-spec/ and syntax (previously
JavaRosa / OpenRosa). I love the declarative style of the spec. I searched
for any turing complete declarative languages (which should have loops),
and found XLST (XLST Language https://en.wikipedia.org/wiki/XSLT, XLST
Elements https://en.wikipedia.org/wiki/XSLT_elements).

If anyone happens to know any way to accomplish a loop as described using
ODK, I will most certainly implement it. Otherwise it looks like we may
have to use some kind of workaround.

--
You received this message because you are subscribed to the Google Groups
"ODK Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to opendatakit-developers+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--
Mitch Sundt
Software Engineer
University of Washington
mitchellsundt@gmail.com

--
Mitch Sundt
Software Engineer
University of Washington
mitchellsundt@gmail.com

[Joseph_E_Flack_IV]

Hello Mitch,

Thanks for all of your helpful feedback. It appears to me that in ODK 2.0,
we may be able to achieve what we need by either (a) manipulating the
javascript of the form client, or (b) having each instance of what would
have been an ODK 1.0 repeat, be a different form instance which accesses 1
or more shared tables.

It does not appear it's possible to have loop syntax in a spreadsheet
definition.

Just for a bit of background, the question workflow we're looking for is
something like this:

Birth History

1. What is the name of your last child born?
2. When were they born?
   3+ (more questions about child)

Question 1 would be slightly different for children after the first,
referencing something like "...born before ${previous_child_name}?"

Termination History
Similar question workflow going back in time over a number of years, asking
questions about any terminated births.

Contraceptive Use History
Similar question workflow going back in time over a number of years, asking
about contraceptive use or lack thereof.

---

What do you think is the best way to do this in ODK 2.0? It seems to me
that maybe instead of customizing a specific survey client, it might be
better to create 3 tables (birth_history, termination_history,
contraceptive_use_history). Each period of pregnancy or non-pregnancy would
be its own row within one of these tables. Each period would be a new
instance of one of 3 forms (births, terminations, contraceptive use).

We would start with a questionnaire directed to the respondent leading up
to these types of questions. Then I am assuming that when the 'birth
questions' are ready to start, it would close that form and start a 1st
instance of a birth history form. And for each period of history, they
would fill out a new form. Perhaps the last question would be "did you have
any other children?" If the answer is no, it would close that form and open
up a termination form. These would repeat in the same manner. Then the
contraceptive use history. Then on the final form instance of contraceptive
use history, that form would close and re-open the initial questionnaire,
hopefully going to the point where they left off.

During all instances of each of these 3 types of forms, constraint logic
would query the 3 database table and make sure that there was no previously
entered period which conflicts with the period currently being entered.

This seems a bit cumbersome. But if it works fluidly, I can definitely see
the advantage. Am I on the right track, or am I quite off on how we would
use ODK 2.0 to collect this kind of data?

[Mitch_S]

Yes, this is how I would approach it, with a main survey about the
respondent, then 3 tables for each of birth, termination and contraceptive
histories, with the instanceID of the respondent survey stored as a
pre-populated field (that you have defined) in each of those tables (so you
can link them back to the respondent).

···

On Sun, Nov 13, 2016 at 6:01 PM, Joseph E. Flack IV wrote:

Hello Mitch,

Thanks for all of your helpful feedback. It appears to me that in ODK 2.0,
we may be able to achieve what we need by either (a) manipulating the
javascript of the form client, or (b) having each instance of what would
have been an ODK 1.0 repeat, be a different form instance which accesses 1
or more shared tables.

It does not appear it's possible to have loop syntax in a spreadsheet
definition.

Just for a bit of background, the question workflow we're looking for is
something like this:

Birth History

1. What is the name of your last child born?
2. When were they born?
   3+ (more questions about child)

Question 1 would be slightly different for children after the first,
referencing something like "...born before ${previous_child_name}?"

Termination History
Similar question workflow going back in time over a number of years,
asking questions about any terminated births.

Contraceptive Use History
Similar question workflow going back in time over a number of years,
asking about contraceptive use or lack thereof.

---

What do you think is the best way to do this in ODK 2.0? It seems to me
that maybe instead of customizing a specific survey client, it might be
better to create 3 tables (birth_history, termination_history,
contraceptive_use_history). Each period of pregnancy or non-pregnancy would
be its own row within one of these tables. Each period would be a new
instance of one of 3 forms (births, terminations, contraceptive use).

We would start with a questionnaire directed to the respondent leading up
to these types of questions. Then I am assuming that when the 'birth
questions' are ready to start, it would close that form and start a 1st
instance of a birth history form. And for each period of history, they
would fill out a new form. Perhaps the last question would be "did you have
any other children?" If the answer is no, it would close that form and open
up a termination form. These would repeat in the same manner. Then the
contraceptive use history. Then on the final form instance of contraceptive
use history, that form would close and re-open the initial questionnaire,
hopefully going to the point where they left off.

During all instances of each of these 3 types of forms, constraint logic
would query the 3 database table and make sure that there was no previously
entered period which conflicts with the period currently being entered.

This seems a bit cumbersome. But if it works fluidly, I can definitely see
the advantage. Am I on the right track, or am I quite off on how we would
use ODK 2.0 to collect this kind of data?

--
You received this message because you are subscribed to the Google Groups
"ODK Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to opendatakit-developers+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--
Mitch Sundt
Software Engineer
University of Washington
mitchellsundt@gmail.com

[Joseph_E_Flack_IV]

Thanks for that clarification, Mitch.

That is interesting. That is so much different than ODK 1.0, I wasn't sure
if I was on the right track. I'll discuss what I've learned with my
organization. Hopefully ODK 2.0 can solve many of the difficult use cases
we're experiencing. Thanks!