Calling external databases

Hi,

I was wondering if ODK is able to call external databases to define
constraints or parameters for a question or to autofill cascading
dropdown menus.

For example, if I selected a certain province, only the cities in
those provinces would be able to be selected.

Or if I was searching through occupations, I could write in "engineer"
and it would show the different types of engineers for me to select
from.

If it is possible, how difficult would it be to implement this in ODK.

Thanks!

To add to Lena's question (or actually take it in a slightly different but
complementary direction), can ODK call up external databases for single
questions?

A few other use cases:

  1. Load a long list of options (unconditionally). Consider an
    agricultural survey where a household must report the crop(s) it plants.
    Coding in all possible crops as multiple-choice options could be
    time-consuming and perhaps error-prone. Could ODK call up an external
    database of crops and allow users to select (for simplicity of this example)
    a single listed option?
  2. Reference a database of constraints. Consider a health survey that
    gathers anthropometric data (e.g., age height, weight, BMI, etc.).
    Different child ages have different associated ranges of acceptable
    anthropometric measures. Rather than code this, could ODK call an external
    database of constraints, such that age drives the lookup of age-relevant
    constraints? (Why a database of constraints? Here I am assuming it would be
    easier to code, and perhaps a more versatile set-up if the same survey is
    used across several countries, as the potential norms could vary slightly
    from one setting to the next.) If so, could you give an idea of how someone
    would code the back-end "interface" between XForms and an external database
    of constraints in a different format?

Best,
Arthur

··· On Mon, Nov 22, 2010 at 4:26 PM, Lena N wrote:

Hi,

I was wondering if ODK is able to call external databases to define
constraints or parameters for a question or to autofill cascading
dropdown menus.

For example, if I selected a certain province, only the cities in
those provinces would be able to be selected.

Or if I was searching through occupations, I could write in "engineer"
and it would show the different types of engineers for me to select
from.

If it is possible, how difficult would it be to implement this in ODK.

Thanks!

--
Post: opendatakit@googlegroups.com
Unsubscribe: opendatakit+unsubscribe@googlegroups.comopendatakit%2Bunsubscribe@googlegroups.com
Options: http://groups.google.com/group/opendatakit?hl=en

these are all features we are making steady (but slow) progress towards.

there is now basic support in the javarosa core for itemsets. that
functionality will allow you do things like get a list of countries
and drill down to districts and then down to cities. i expect that
functionality will be in the next release of collect. that list will
have to be coded into the form for now.

filtering through a very long list of items is doable today with the
android widgets. it'd be a nice to have, but it's not at the top of
our todo list. good news is it's pretty easy to code this up and check
it into trunk. it'd be very similar to what we do in odk clinic with
patient lists...

we have an outline on how to connect to external systems at
http://code.google.com/p/opendatakit/wiki/XFormControlExtensionProposal.
that should enable people to build their own widgets and connect them
to external apps, dbs, etc. as always, if anyone has the dev cycles to
build this into the core, we need help :slight_smile:

··· On Mon, Nov 22, 2010 at 13:45, Arthur Shaw wrote: > To add to Lena's question (or actually take it in a slightly different but > complementary direction), can ODK call up external databases for single > questions? > A few other use cases: > > Load a long list of options (unconditionally). Consider an agricultural > survey where a household must report the crop(s) it plants. Coding in all > possible crops as multiple-choice options could be time-consuming and > perhaps error-prone. Could ODK call up an external database of crops and > allow users to select (for simplicity of this example) a single listed > option? > Reference a database of constraints. Consider a health survey that gathers > anthropometric data (e.g., age height, weight, BMI, etc.). Different child > ages have different associated ranges of acceptable anthropometric measures. > Rather than code this, could ODK call an external database of constraints, > such that age drives the lookup of age-relevant constraints? (Why a > database of constraints? Here I am assuming it would be easier to code, and > perhaps a more versatile set-up if the same survey is used across several > countries, as the potential norms could vary slightly from one setting to > the next.) If so, could you give an idea of how someone would code the > back-end "interface" between XForms and an external database of constraints > in a different format? > > Best, > Arthur > On Mon, Nov 22, 2010 at 4:26 PM, Lena N wrote: >> >> Hi, >> >> I was wondering if ODK is able to call external databases to define >> constraints or parameters for a question or to autofill cascading >> dropdown menus. >> >> For example, if I selected a certain province, only the cities in >> those provinces would be able to be selected. >> >> Or if I was searching through occupations, I could write in "engineer" >> and it would show the different types of engineers for me to select >> from. >> >> If it is possible, how difficult would it be to implement this in ODK. >> >> Thanks! >> >> -- >> Post: opendatakit@googlegroups.com >> Unsubscribe: opendatakit+unsubscribe@googlegroups.com >> Options: http://groups.google.com/group/opendatakit?hl=en > > -- > Post: opendatakit@googlegroups.com > Unsubscribe: opendatakit+unsubscribe@googlegroups.com > Options: http://groups.google.com/group/opendatakit?hl=en >

The currently supported mechanism for embedded itemsets in an XForm is a bit
of a kludge. It places the static data for list population within the form
being
processed, then collects data and places it in a nested group within the
model,
and submits just that nested group. This requires some changes to the way
Aggregate and Collect handle the XForm, since the whole model will no longer

be transmitted, but just a portion of it. So this won't be supported on
anything
prior to Collect 1.1.6 and Aggregate 1.0

A cleaner way of supporting embedded itemsets is to have multiple model
instances within the XForm, and to be able to reference the alternate
itemsets
when populating data into the form. This is also the way to support
external
databases.

This required requires changes to the javarosa core. Embedded itemsets
are very close to being supported, but require exposure of named
FormDefinition
objects and their use in the EvaluationContext. It is largely plumbing.

External database support is more complex. The javarosa XForms parser
over-zealously collapses XPath expressions into concrete TreeReference
types. This needs to be carefully teased apart.

To keep the XForm declarative, there will be no way to make changes
to these named form definitions (embedded itemsets or external databases).

These are highly desired changes, and as Yaw says, we'd love folks to
eithet contribute the dev cycles to get them done, or pool funding to pay
Dimagi to prioritize these changes to the javarosa core.

Mitch

··· On Mon, Nov 22, 2010 at 7:56 PM, Yaw Anokwa wrote:

these are all features we are making steady (but slow) progress towards.

there is now basic support in the javarosa core for itemsets. that
functionality will allow you do things like get a list of countries
and drill down to districts and then down to cities. i expect that
functionality will be in the next release of collect. that list will
have to be coded into the form for now.

filtering through a very long list of items is doable today with the
android widgets. it'd be a nice to have, but it's not at the top of
our todo list. good news is it's pretty easy to code this up and check
it into trunk. it'd be very similar to what we do in odk clinic with
patient lists...

we have an outline on how to connect to external systems at
http://code.google.com/p/opendatakit/wiki/XFormControlExtensionProposal.
that should enable people to build their own widgets and connect them
to external apps, dbs, etc. as always, if anyone has the dev cycles to
build this into the core, we need help :slight_smile:

On Mon, Nov 22, 2010 at 13:45, Arthur Shaw jashaw2002@gmail.com wrote:

To add to Lena's question (or actually take it in a slightly different
but
complementary direction), can ODK call up external databases for single
questions?
A few other use cases:

Load a long list of options (unconditionally). Consider an agricultural
survey where a household must report the crop(s) it plants. Coding in
all
possible crops as multiple-choice options could be time-consuming and
perhaps error-prone. Could ODK call up an external database of crops and
allow users to select (for simplicity of this example) a single listed
option?
Reference a database of constraints. Consider a health survey that
gathers
anthropometric data (e.g., age height, weight, BMI, etc.). Different
child
ages have different associated ranges of acceptable anthropometric
measures.
Rather than code this, could ODK call an external database of
constraints,
such that age drives the lookup of age-relevant constraints? (Why a
database of constraints? Here I am assuming it would be easier to code,
and
perhaps a more versatile set-up if the same survey is used across several
countries, as the potential norms could vary slightly from one setting to
the next.) If so, could you give an idea of how someone would code the
back-end "interface" between XForms and an external database of
constraints
in a different format?

Best,
Arthur
On Mon, Nov 22, 2010 at 4:26 PM, Lena N lena.nguyen4@gmail.com wrote:

Hi,

I was wondering if ODK is able to call external databases to define
constraints or parameters for a question or to autofill cascading
dropdown menus.

For example, if I selected a certain province, only the cities in
those provinces would be able to be selected.

Or if I was searching through occupations, I could write in "engineer"
and it would show the different types of engineers for me to select
from.

If it is possible, how difficult would it be to implement this in ODK.

Thanks!

--
Post: opendatakit@googlegroups.com
Unsubscribe: opendatakit+unsubscribe@googlegroups.comopendatakit%2Bunsubscribe@googlegroups.com
Options: http://groups.google.com/group/opendatakit?hl=en

--
Post: opendatakit@googlegroups.com
Unsubscribe: opendatakit+unsubscribe@googlegroups.comopendatakit%2Bunsubscribe@googlegroups.com
Options: http://groups.google.com/group/opendatakit?hl=en

--
Post: opendatakit@googlegroups.com
Unsubscribe: opendatakit+unsubscribe@googlegroups.comopendatakit%2Bunsubscribe@googlegroups.com
Options: http://groups.google.com/group/opendatakit?hl=en

--
Mitch Sundt
Software Engineer
http://www.OpenDataKit.org
University of Washington
mitchellsundt@gmail.com

hi,

There is a thread in the javarosa-developers group
(http://groups.google.com/group/javarosa-developers) with subject
"about dynamic itemsets" with some suggestions about how to achieve
that. I faced the same issue a few months ago and gave up trying to
implement it in the 'right way' (modifications too big, time too
short).

One of the main issues was that javarosa doesn't support yet multiple
instance nodes, and is not possible to refer to an external DB and to
the form model within the same instance node. As Yaw says, connect to
external systems is another issue.

instead, I took the shortcut of inserting the values directly into the
form template, before loading it. It's definitely not a good solution
if the form is vastly used and the values are changing, because each
template is different and ODK generates a .formdef file per instance,
which is stored in the sdcard. But for our case is working fine, so
far.

Data is queried to the DB from an external Activity and provided to
FormEntryActivity through Intent.putExtra(). An ODK's customized
version reads the Intent.getExtras() and builds the itemsets if
required, before loading the form.

sorry for suggesting bad practices, but if someone is in a hurry, that
could help :wink:

And one question to ODK's team: are these 'dev cycles' being discussed
somewhere?

best,

pau.

··· 2010/11/23 Mitch Sundt : > The currently supported mechanism for embedded itemsets in an XForm is a bit > of a kludge. It places the static data for list population within the form > being > processed, then collects data and places it in a nested group within the > model, > and submits just that nested group. This requires some changes to the way > Aggregate and Collect handle the XForm, since the whole model will no longer > be transmitted, but just a portion of it. So this won't be supported on > anything > prior to Collect 1.1.6 and Aggregate 1.0 > > A cleaner way of supporting embedded itemsets is to have multiple model > instances within the XForm, and to be able to reference the alternate > itemsets > when populating data into the form. This is also the way to support > external > databases. > > This required requires changes to the javarosa core. Embedded itemsets > are very close to being supported, but require exposure of named > FormDefinition > objects and their use in the EvaluationContext. It is largely plumbing. > > External database support is more complex. The javarosa XForms parser > over-zealously collapses XPath expressions into concrete TreeReference > types. This needs to be carefully teased apart. > > To keep the XForm declarative, there will be no way to make changes > to these named form definitions (embedded itemsets or external databases). > > These are highly desired changes, and as Yaw says, we'd love folks to > eithet contribute the dev cycles to get them done, or pool funding to pay > Dimagi to prioritize these changes to the javarosa core. > > Mitch > > On Mon, Nov 22, 2010 at 7:56 PM, Yaw Anokwa wrote: >> >> these are all features we are making steady (but slow) progress towards. >> >> there is now basic support in the javarosa core for itemsets. that >> functionality will allow you do things like get a list of countries >> and drill down to districts and then down to cities. i expect that >> functionality will be in the next release of collect. that list will >> have to be coded into the form for now. >> >> filtering through a very long list of items is doable today with the >> android widgets. it'd be a nice to have, but it's not at the top of >> our todo list. good news is it's pretty easy to code this up and check >> it into trunk. it'd be very similar to what we do in odk clinic with >> patient lists... >> >> we have an outline on how to connect to external systems at >> http://code.google.com/p/opendatakit/wiki/XFormControlExtensionProposal. >> that should enable people to build their own widgets and connect them >> to external apps, dbs, etc. as always, if anyone has the dev cycles to >> build this into the core, we need help :) >> >> On Mon, Nov 22, 2010 at 13:45, Arthur Shaw wrote: >> > To add to Lena's question (or actually take it in a slightly different >> > but >> > complementary direction), can ODK call up external databases for single >> > questions? >> > A few other use cases: >> > >> > Load a long list of options (unconditionally). Consider an agricultural >> > survey where a household must report the crop(s) it plants. Coding in >> > all >> > possible crops as multiple-choice options could be time-consuming and >> > perhaps error-prone. Could ODK call up an external database of crops >> > and >> > allow users to select (for simplicity of this example) a single listed >> > option? >> > Reference a database of constraints. Consider a health survey that >> > gathers >> > anthropometric data (e.g., age height, weight, BMI, etc.). Different >> > child >> > ages have different associated ranges of acceptable anthropometric >> > measures. >> > Rather than code this, could ODK call an external database of >> > constraints, >> > such that age drives the lookup of age-relevant constraints? (Why a >> > database of constraints? Here I am assuming it would be easier to code, >> > and >> > perhaps a more versatile set-up if the same survey is used across >> > several >> > countries, as the potential norms could vary slightly from one setting >> > to >> > the next.) If so, could you give an idea of how someone would code the >> > back-end "interface" between XForms and an external database of >> > constraints >> > in a different format? >> > >> > Best, >> > Arthur >> > On Mon, Nov 22, 2010 at 4:26 PM, Lena N wrote: >> >> >> >> Hi, >> >> >> >> I was wondering if ODK is able to call external databases to define >> >> constraints or parameters for a question or to autofill cascading >> >> dropdown menus. >> >> >> >> For example, if I selected a certain province, only the cities in >> >> those provinces would be able to be selected. >> >> >> >> Or if I was searching through occupations, I could write in "engineer" >> >> and it would show the different types of engineers for me to select >> >> from. >> >> >> >> If it is possible, how difficult would it be to implement this in ODK. >> >> >> >> Thanks! >> >> >> >> -- >> >> Post: opendatakit@googlegroups.com >> >> Unsubscribe: opendatakit+unsubscribe@googlegroups.com >> >> Options: http://groups.google.com/group/opendatakit?hl=en >> > >> > -- >> > Post: opendatakit@googlegroups.com >> > Unsubscribe: opendatakit+unsubscribe@googlegroups.com >> > Options: http://groups.google.com/group/opendatakit?hl=en >> > >> >> -- >> Post: opendatakit@googlegroups.com >> Unsubscribe: opendatakit+unsubscribe@googlegroups.com >> Options: http://groups.google.com/group/opendatakit?hl=en > > > > -- > Mitch Sundt > Software Engineer > http://www.OpenDataKit.org > University of Washington > mitchellsundt@gmail.com > > -- > Post: opendatakit@googlegroups.com > Unsubscribe: opendatakit+unsubscribe@googlegroups.com > Options: http://groups.google.com/group/opendatakit?hl=en >

i don't understand what you mean, pau...

··· On Tue, Nov 23, 2010 at 11:57, Pau Varela wrote: > And one question to ODK's team: are these 'dev cycles' being discussed > somewhere? >

hi Yaw,

in this thread, both you and Mitch mentioned some 'dev cycles' and the
fact that you need help with them. Aren't they 'development cycles'?
and if so, how to help? this was basically my question, to know
whether there is a discussion about these 'dev cycles' and what needs
to be done, in order to see how to contribute..

thanks,

pau.

··· 2010/11/26 Yaw Anokwa : > i don't understand what you mean, pau... > > On Tue, Nov 23, 2010 at 11:57, Pau Varela wrote: >> And one question to ODK's team: are these 'dev cycles' being discussed >> somewhere? >> > > -- > Post: opendatakit@googlegroups.com > Unsubscribe: opendatakit+unsubscribe@googlegroups.com > Options: http://groups.google.com/group/opendatakit?hl=en >

pau,

we have a roadmap at http://code.google.com/p/opendatakit/wiki/Roadmap
and bugs at http://code.google.com/p/opendatakit/issues/list that
lists then things we are working on. if you can help with any of
those, that'd be great. if you want more broad impact, the javarosa
library we use also has lots of open problems listed at
http://bitbucket.org/javarosa/javarosa/issues. beyond code, we just
need help with everything!

great at design? maybe you can help us with user interfaces.

love making videos? maybe you can do a screencast of our tools.

natural leader? maybe you can organize a hackathon.

speak fluent swahili? maybe you can translate odk aggregate.

experienced tester? maybe you can triage bugs.

excel at writing? maybe you can keep our wiki easy to read.

the list goes on and on :slight_smile:

yaw

··· On Fri, Nov 26, 2010 at 01:49, Pau Varela wrote: > hi Yaw, > > in this thread, both you and Mitch mentioned some 'dev cycles' and the > fact that you need help with them. Aren't they 'development cycles'? > and if so, how to help? this was basically my question, to know > whether there is a discussion about these 'dev cycles' and what needs > to be done, in order to see how to contribute.. > > thanks, > > pau. > > 2010/11/26 Yaw Anokwa : >> i don't understand what you mean, pau... >> >> On Tue, Nov 23, 2010 at 11:57, Pau Varela wrote: >>> And one question to ODK's team: are these 'dev cycles' being discussed >>> somewhere? >>> >> >> -- >> Post: opendatakit@googlegroups.com >> Unsubscribe: opendatakit+unsubscribe@googlegroups.com >> Options: http://groups.google.com/group/opendatakit?hl=en >> > > -- > Post: opendatakit@googlegroups.com > Unsubscribe: opendatakit+unsubscribe@googlegroups.com > Options: http://groups.google.com/group/opendatakit?hl=en >