Hi Matt,
Thanks for your reply.
The search() query examples in that example form were just meant to
experiment with different syntax options, so they actually don't make much
sense as actual queries; the case where I'm querying district using a
village name was somewhat nonsensical. The idea is that you'd be able to
search any .csv column for any string value, and we're implementing four
types of search queries: exact, contains, starts-with, and ends-with. The
syntax is still in flux as we work on the implementation. (We're now adding
a "filter" option, for example, so that you can search within a subset of
the .csv data.)
Our pulldata() support has already been released, and you're right that it
only allows you to pull one field at a time. That's a definite drawback in
cases where you're pulling lots of data from the .csv, and the kind of
dict-based approach you suggest could make a lot of sense.
You're also right that, as things stand, you have to upload .csv files
with every form, so a URL reference would be better for cases where the
same .csv is shared across multiple forms. However, then there would need
to be a separate system of maintaining/updating .csv files. With the .csv
files as part of the form, they essentially piggy-back onto the existing
form versioning system, and it's relatively easy to keep track of the fact
that .csv files basically come with the form (and you can download a new
form definition to update the .csv files).
Obviously, ODK 2.0 has a much richer method of flexibly dealing with
different data flows, and our intent was not so much to create an
alternative approach. Rather, we simply wanted to provide for some basic
ability to deal with pre-loaded data in the short-run since it seemed like
such a great and urgent need for many users. Our approach has been to try
and fit within the existing ODK 1.0 architecture to the greatest extent
possible, and to just work the basic pre-loading support in with the most
graceful methods we can manage.
Generally, we've been trying to do things with the minimum necessary
changes to ODK code, conditional on providing a decent interface for users.
That's why, for example, we have built a series of "expression builders"
within the SurveyCTO web interface, to make it as easy as possible for
users while sticking, for the most part, with the existing XLSForm
implementation. So, for instance, "Pull field from pre-loaded .csv file" is
an option in our "calculation builder," which helps then construct the
proper pulldata() call.
These new features are part of our custom build of Collect, and we have
associated branches for pyxform, Aggregate, Briefcase, etc. While we've
been a bit renegade and non-consultative in many of the features we've been
developing, it's been our policy to share "core changes" back to the
community whenever they seem useful and general enough. Thus, for example,
our fix to allow calls to "position(..)" to get the current repeat group
index, and anything we might do to extend the functionality with respect to
calling external apps. We're definitely interested in helping to improve
the open-source foundations as we go along, though also we plan to keep
blazing ahead with new features layered on top, as demanded by our
customers.
Because ODK 2.0 will be so different -- and because it will handle
pre-loading use cases in its own way -- I had assumed that our pre-loading
support would be something of a short-term bridge. Assuming that's the
case, it doesn't make sense to think too hard about, e.g., the best
long-term way to integrate it with XLSForm. However, it does seem to make
sense to think hard about better ways of integrating with external apps in
general, since that does seem to be a long-term issue.
Also, I should really engage more on 2.0 to make sure that the use cases
we're addressing with our current pre-load support are, indeed, gracefully
handled in ODK 2.0. These are cases where you have existing data from a
baseline survey, listing survey, or administrative data, and you want to
reference it easily in your forms. In these cases, you don't need the data
to be editable, you just need to be able to pull it in and reference it. I
actually don't know how easy that will be in the 2.0 ecosystem, but ideally
it will be extremely easy.
Thanks again,
Chris
On Thu, May 16, 2013 at 9:44 AM, Matt Berg mlberg@gmail.com wrote:
Chris,
Copying in members of our team that work on XLSForm as they could speak
to the issue of the XLSForm to me.
Just some background, we basically try and coordinate/discuss changes on
the xlsform spec between Gaetano / Nathan (copied) and our team. It sounds
like you're getting involved in a meaningful way which is great too. Key
is justing getting people's feedback so we can all try and implement it in
a sane way. We've made some proposed SMS changes we need to introduce to
the group too.
Chris - so just to clarify the csv is downloaded as a file and ODK
Collect has been modified to store those values in a table that are
accessible for that form? The reason I am wondering if we can support a URL
reference in stead of media file is it would make it a lot easier to
support dynamic lists on the server (without doing a hackly reference link
to the media url for the form) it would also allow us to share a csv across
multiple forms which I think would probably be very useful. Right now you
would have to upload the same csv to each form right?
So in terms of logic, let me try and reexplain for my own understanding.
The form you shared shows off two different functionalities from what I can
tell.
First, there is the dynamic select which allows us to define a large
cascading select options from the hhplotselects.csv.
The first select we pass the district: search('hhplotselects',
'district', ${hhiddistrict})
which looks like is smart enough that we don't need to pass the region.
Would* search('hhplotselects','region',${hhregion}) *also work?
I'm confused by the next select though:
search('hhplotselects', 'district', ${hhidvillage})
Is this doing a district lookup based on the village you are in? Ie) this
is meant to return a district and not a village? Is searching one branch
down into the cascade supported?
The other function being introduced is pulldata
pulldata('hhplotdetails', 'plot1description', 'hhid_key', ${hhid})
This works by returning plot1description from hhplotdetails where
hhid_key = ${hhid}
Since we are using a calculate does this limit us to return one property
a time from the sheet?
This makes me wonder if we want to support the idea of dict's in xlsform.
Something like this would be much more elegant.
type: calculate name: plotdetails calculation:
pulldata('hhplotdetails','plot1_desc, plot1_size, plot2_desc,
plot2_name','hhid_id', ${hhid})
We could then reference things in the xlsform later as:
${plotdetails.plot1_desc}
Instead of pull_data should we maybe just call this function get or *
get_data*?
Lastly, this is then used to dynamically name the plot descriptions.
Note: The example is a bit confusing as it looks like a cascade.
Prabhas/Dickson thoughts on the second point about updating multiple
groups?
Chris - lastly is this now supported in ODK Collect or a version you have?
Thanks for putting so much thought into this. This is really good
thinking and I like the proposed changes.
Matt
On Thu, May 16, 2013 at 8:44 AM, Christopher Robert < chrislrobert@gmail.com> wrote:
Hi Matt,
I see that Mitch just added code to pass the current field value along
with the external intent, and implementing
his appearance="ex:intentstring(bundlename1=fieldname1,bundlefieldname2)"
suggestion wouldn't be too difficult. We might need to further change
XLSForm to call insert_xpaths() on all appearance column values, rather
than on just those associated with select fields. But that seems like it
would be harmless. That way, any ${fieldname} references would be
translated into full XPaths, which would be easier for Collect to process.
That wouldn't solve the broader issue, though, of being able to return
multiple values to update multiple fields in a group. We don't actually
have a need for that ourselves at the moment, but if we're in there
revising the code, maybe we can try to tackle that too. Would that be your
preferred interface (groups with multiple fields being updated) for your
nearby-points widget?
Chris
On Wed, May 15, 2013 at 3:48 PM, Matt Berg mlberg@gmail.com wrote:
Chris,
Thanks for the explanation. Very helpful.
Can you explain a bit more what you mean by resolving the ${fieldname}
references?
For the CSV lists, I think it would be ideal if this could point to an
authenticated csv endpoint on the server. Then during the sync it would
download a cached version of the csv. That would allow you to point to
external sources ultimately if we implemented something like etag support.
So in the options tab are you putting the link to the csv in appearance
and living the options blank? Could you maybe share a sample of what
you're thinking? Also, what's the thoughts on handling cascades?
As for the external widget, we're very interested in developing a
widget that would allow you to lookup points close to you (based on your
location) and preload that data into your form (in-effect doing a check-in).
Thanks would be really amazing to make progress on this as it's such a
long standing request by the community.
Matt
On Wed, May 15, 2013 at 4:23 PM, Christopher Robert < chrislrobert@gmail.com> wrote:
Hi Matt,
One thing that we want to do, now that we're able to load .csv data
into calculate fields, is allow the choice lists for select and select1
prompts to be pulled from .csv data as well. We've been fiddling around
with the best way to do that, and at the moment it seems like the most
graceful approach is to specify the parameters for that in the select_one
or select_multiple appearance column. Thus, you can specify any of the
existing appearance tags to govern the UI style, plus you can indicate that
the labels and values should be pulled from a pre-loaded .csv, via a query.
The only change we needed to make to XLSForm was to resolve ${fieldname}
references within those appearances, which was easy to do.
For the existing multiple-choice UI options, this seems like the most
seamless integration for users. While we were originally planning to use an
external app, that would require extra button-clicks, as you wouldn't be
able to just swipe along the way you do with normal, internal prompts.
For more complex search-and-select operations, though, it does seem
like an external app is a better way to go. Certainly, it would require
less invasive changes to the ODK codebase. So, for additional UI
functionality, we're still thinking about adding one or more external
widgets. For those, we can go with the "group" approach Mitch suggests,
and/or some lighter method of specifying parameters to the existing
ex:intent functionality.
If you have any suggestions for graceful ways that this might be
worked into XLSForm, we're definitely all ears. It's all well and good to
provide neat new functionality for users, but if it's not easy to use, it
will be a real headache for everybody.
Thanks,
Chris
On Wed, May 15, 2013 at 3:09 PM, Matt Berg mlberg@gmail.com wrote:
Chris,
This sounds very exciting and something we want to support.
Will this make require any modifications in the xlsform spec? If so,
can you let us know what you're thinking?
Thanks,
Matt
On Wed, May 15, 2013 at 4:06 PM, Christopher Robert < chrislrobert@gmail.com> wrote:
Mitch,
Thanks for the additional thoughts. We're working on this now in the
context of our new dynamic-search-and-select widgets, and we'll see if we
can kick something useful back to the main stream.
Thanks again!
Chris
On Wed, May 15, 2013 at 8:23 AM, Mitch Sundt < mitchellsundt@gmail.com> wrote:
I.e., I would still declare form fields for the values passed to
the 3rd party app
And then have a parameter mapping after the intent string - perhaps
surrounded by parentheses, e.g.,
appearance="ex:intentstring(bundlename1=fieldname1,bundlefieldname2)"
Here, we send a bundle of (bundlename1,bundlefieldname2) to the 3rd
party app using the values and data types of (fieldname1,bundlefieldname2)
in the form.
The additional separate calculated or fixed-value fields take up
space in the form, but they avoid having to parse, interpret and evaluate
an expression in an attribute associated with a body prompt (appearance
attribute), which as you note would be hard to associate with a proper
evaluation context.
Mitch
On May 14, 2013 6:32 PM, "Mitch Sundt" mitchellsundt@gmail.com wrote:
The group widget could be smart about updates. E.g., if the fields
were (calculate) fields that were read-only, it could skip the update
attempt.
We can certainly add additional 'send this' attribute processing
to the string et al widgets.
On May 14, 2013 1:54 AM, "Christopher Robert" < chrislrobert@gmail.com> wrote:
Hi Mitch,
Thanks for your reply.
I think that I understand your proposed mechanism, and it makes a
lot of sense for cases where you want an external app to return more than
one value (i.e., a bundle). However, I am somewhat less convinced about how
straightforward it is for the simple case where you want to pass one or
more parameters to an external app, and just get a single return value
back. ExStringWidget and friends are almost perfect, in this regard, except
for the fact that there is no mechanism to supply parameters.
If we were to implement our current widgets this way (essentially
just select_one and select_multiple variants that use pre-loaded data and a
query to filter), then we would have to have users add extra groups and
fields into their forms just to pass required parameters. These would
generally be calculate fields, but then, given the proposal, the widget
would try to update those calculate fields again with values returned by
the widget. Conceivably, we could build loads of logic into XLSForm to
auto-generate the groups and fields, but it doesn't seem like a great fit
for the simple parameterized-call-with-single-return-value case.
Again, ExStringWidget and friends are very close to what we want, but for
their inability to take parameters.
But perhaps I'm missing some element of your proposal? I'd
certainly like to implement this in a way that fits well with the overall
ODK plan, if at all possible.
Thanks again,
Chris
On Mon, May 13, 2013 at 5:55 PM, Mitch Sundt < mitchellsundt@gmail.com> wrote:
I would recommend instead following this:
http://code.google.com/p/opendatakit/wiki/XFormControlExtensionProposal
If you change ODK Collect so that a group of values is sent to
the 3rd party app and then returned, it would be more in keeping, and would
avoid introducing additional mechanisms to pass XML attributes into and out
of the 3rd party app.
I.e, on entering the group screen, you would have a button to
launch the 3rd party app.
When launching the app, the values for all the fields in the
group are sent to the app.
On return, the result intent is then used to retrieve and update
all the fields within the group.
Mitch
On Sun, May 12, 2013 at 10:19 PM, Christopher Robert < crobert@surveycto.com> wrote:
All,
The ability to call external apps to retrieve form data is
extremely helpful, but we would like to be able to pass parameters to these
external apps. It doesn't look like any such support is presently there,
but it doesn't look too hard to add. Parameters could be included as the
"data" for the intent, or as the "extras." You could easily add parameters
into the existing appearance like so:
ex:intent.path:parameter-string
The one tricky thing is that we'd want parameters to be able to
include references to dynamic form data (i.e., evaluated XPath
expressions). In principle, we could call the XPath expression evaluator on
whatever was after the second colon, so that you could have:
ex:intent.path:'literal string'
ex:intent.path:functioncall(...)
etc.
The one tricky thing there is that we don't have an
EvaluationContext there in Ex...Widget.java, and I'm not sure how hard it
would be to scare up such a context.
We can do all of this as custom mods to SurveyCTO, but it seems
like it might be a useful addition to ODK. Would the ODK team be interested
in us contributing such a change? And, if so, any input on its
implementation?
Thanks,
Chris
--
You received this message because you are subscribed to the
Google Groups "ODK Developers" group.
To unsubscribe from this group and stop receiving emails from
it, send an email to
opendatakit-developers+unsubscribe@googlegroups.com.
For more options, visit
https://groups.google.com/groups/opt_out.
--
Mitch Sundt
Software Engineer
University of Washington
mitchellsundt@gmail.com
--
You received this message because you are subscribed to the
Google Groups "ODK Developers" group.
To unsubscribe from this group and stop receiving emails from
it, send an email to
opendatakit-developers+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out
.
--
You received this message because you are subscribed to the
Google Groups "ODK Developers" group.
To unsubscribe from this group and stop receiving emails from it,
send an email to
opendatakit-developers+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.
--
You received this message because you are subscribed to the Google
Groups "ODK Developers" group.
To unsubscribe from this group and stop receiving emails from it,
send an email to
opendatakit-developers+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.
--
You received this message because you are subscribed to the Google
Groups "ODK Developers" group.
To unsubscribe from this group and stop receiving emails from it,
send an email to opendatakit-developers+unsubscribe@googlegroups.com
.
For more options, visit https://groups.google.com/groups/opt_out.
--
You received this message because you are subscribed to the Google
Groups "ODK Developers" group.
To unsubscribe from this group and stop receiving emails from it,
send an email to opendatakit-developers+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.
--
You received this message because you are subscribed to the Google
Groups "ODK Developers" group.
To unsubscribe from this group and stop receiving emails from it, send
an email to opendatakit-developers+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.
--
You received this message because you are subscribed to the Google
Groups "ODK Developers" group.
To unsubscribe from this group and stop receiving emails from it, send
an email to opendatakit-developers+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.
--
You received this message because you are subscribed to the Google
Groups "ODK Developers" group.
To unsubscribe from this group and stop receiving emails from it, send
an email to opendatakit-developers+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.
--
You received this message because you are subscribed to the Google Groups
"ODK Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to opendatakit-developers+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.
--
You received this message because you are subscribed to the Google Groups
"ODK Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to opendatakit-developers+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.