Hi Everyone,
My name is Taylor Downs, and I've been a
big
ODK "fanboy" since I first came across it working for an HIV-prevention
organization in South Africa about five years ago
.
In 2010 I co-founded a consulting company (Vera Solutions) that has since
designed and implemented data systems for more than 160 social impact
organizations around the world. As more and more of these implementations
required
full-featured
cloud-based backends *AND *inexpensive, easy-to-use, offline-capable mobile
tools we found ourselves needing to mix and match technologies. ODK became
our
mobile
data
-
capture workhorse, and Salesforce.com our backend of choice.
The flexibility of ODK (and xForms) makes it powerful, but also difficult
to integrate
—
at times it feels like we're trying to hit a moving target. Our clients
demand complex forms, and the ability to change those forms often as their
programs evolve. Hard-coding integrations for each project was too costly
and time-consuming, so we set out to develop an open-source integration
platform that could be used to quickly and easily connect "source" ODK
submission data (actually, *any *JSON data
—
provided you have prior knowledge of the schema) to Salesforce.com, DHIS2,
and
other "destination" systems.
In short, we're building an open-source Zapier.com or IFTTT.com that can
handle complex data sources with nested repeat blocks and the need to
update or upsert against existing records in all sorts of systems.
This has been a gnarly undertaking, and has prompted a few ideas which I
want to share with you. Questions, feedback, and disagreement welcome!
-
Good systems have interchangeable parts. We may be biased because
Stu, our technical lead, is a service-oriented architecture buff, but the
reality is this: If you're building a complete data system (people,
processes, technologies, feedback loops) and you do a Good Job, the
organization will have access to the information they need to change their
intervention... to make it better. This means new processes, new users, and
likely new technologies. It's tempting to build everything on the same
platform, but often you're sacrificing the needs of a particular
user-group, or future flexibility by not making your system adequately
modular. - The hard part of data integration should be contextual
—knowing how information should work together to provide some benefit
for a bunch of humans—but all too often it is technical. We're hoping that
by knocking down the technical barriers, program managers will be able to
create and manage more useful connections between tools. In the past,
setting up and managing an integration was the domain of the IT team—if
there was one. I think we can make it less "scary". - *Consider letting other parts of your system "write" your ODK forms
for you. *This may not be a novel idea to many of you, but if you're
integrating ODK with another system, you may find that it's easier to have
that destination system generate xForms via a script. If new forms need to
be generated and disseminated based on changes to objects and fields in
Salesforce, or data-elements and data-sets in DHIS2, then why not abstract
this process as it will undoubtedly continue. We've even played around with
this is a way to generate class-specific xForms with Present/Absent
questions for each registered participant. Lots of directions you can go
from here. -
Make use of third-party visualization and communication tools. There
has been an explosion of great mapping, visualization, analytics and even
notification tools. They're all useless without solid source data, and ODK
has, time and time again, proven to be invaluable for us as a way to gather
rich data from mobile phones in disconnected areas. Thinking of ODK as a
source in live, multi-faceted systems, rather than an "end-to-end"
deployment suitable for periodic research and data gathering efforts has
greatly expanded the range of fit across our clients.
I'm
particularly eager to hear what other technologies have proved valuable
to connect to ODK, but am also happy to dig in on any of the above points.
Also, if anyone is interested in OpenFn (the integration platform we're
building now) please check us out on Github https://github.com/OpenFn/.
Below, I've pasted a sample "expression" that's used to map submissions
from a deeply nested form to multiple parent and child objects in
Salesforce just to get you excited! Users will create those expressions
using a little point-and-click builder https://www.openfn.org/v2. The
whole thing is Elixir and Javascript, using React for the front-end. What
you see below takes a "credentials.json" file and published data from ODK's
JSON publisher. It makes use of a 'language-pack' for Salesforce. Anyway, I
hope you get a kick out of it, and Happy Holidays to all!
steps( each( join("$.data.data[]", "$.data.formVersion", "formVersion"),
combine( create("Parent_Object__c", fields( field("ODK_formVersion__c",
sourceValue("$.data.formVersion")), field("ODK_time_end__c", sourceValue("
$.data.end")), field("weather__c", sourceValue("$.data.weather")), field("
ODK_device__c", sourceValue("$.data.deviceid")), field("lat__c", sourceValue
("$.data.geopoint:Latitude")), field("lon__c", sourceValue("
$.data.geopoint:Longitude")), field("ODK_uuid__c", sourceValue("
$.data.meta-instance-id")), field("landing_site__c", sourceValue("
$.data.landing_site")) )), each( join("$.data.rpt_boats[]", "
$.references[0].id", "parentId"), combine( create("Boat__c", fields( field("
parent__c", sourceValue("$.data.parentId")), field("reg_type__c",
sourceValue("$.data.reg_type")), field("comment_image_url_long__c",
sourceValue("$.data.comment_image")), field("engine_capacity__c",
sourceValue("$.data.engine_capacity")), field("catch_specie_list__c",
function(state) { return Array.apply( null, sourceValue("
$.data.catch_specie_list")(state) ).join(', ') }) )), each( join("
$.data.rpt_catch[]", "$.references[0].id", "Id"), combine( create("Catch__c
", fields( field("parent__c", sourceValue("$.data.Id")), lookup("
classification_record__r", "name", "$.data.selected_specie"), field("
catch_bait_manual__c", sourceValue("$.data.catch_bait_manual")) )), each(
join("$.data.rpt_sample[]", "$.references[0].id", "Id"), create("
CCM_Sample__c", fields( field("parent__c", sourceValue("$.data.Id")), field(
"sample_berry__c", sourceValue("$.data.sample_berry")), )) ) ) ) )) ) ))