Discussion Series: Using ODK as a data source for DHIS2 and Salesforce

Hi Everyone,

My name is Taylor Downs, and I've been a
​ big
ODK "fanboy" since I first came across it working for an HIV-prevention
organization in South Africa about five years ago
In 2010 I co-founded a consulting company (Vera Solutions) that has since
designed and implemented data systems for more than 160 social impact
organizations around the world. As more and more of these implementations
​ ​
cloud-based backends *AND *inexpensive, easy-to-use, offline-capable mobile
tools we found ourselves needing to mix and match technologies. ODK became
capture workhorse, and Salesforce.com our backend of choice.

The flexibility of ODK (and xForms) makes it powerful, but also difficult
to integrate
at times it feels like we're trying to hit a moving target. Our clients
demand complex forms, and the ability to change those forms often as their
programs evolve. Hard-coding integrations for each project was too costly
and time-consuming, so we set out to develop an open-source integration
platform that could be used to quickly and easily connect "source" ODK
submission data (actually, *any *JSON data
provided you have prior knowledge of the schema) to Salesforce.com, DHIS2,
other "destination" systems.

​In short, we're building an open-source Zapier.com or IFTTT.com that can
handle complex data sources with nested repeat blocks and the need to
update or upsert against existing records in all sorts of systems.​
​ This has been a gnarly undertaking, and has prompted a few ideas which I
want to share with you. Questions, feedback, and disagreement welcome!

  1. Good systems have interchangeable parts. We may be biased because
    Stu, our technical lead, is a service-oriented architecture buff, but the
    reality is this: If you're building a complete data system (people,
    processes, technologies, feedback loops) and you do a Good Job, the
    organization will have access to the information they need to change their
    intervention... to make it better. This means new processes, new users, and
    likely new technologies. It's tempting to build everything on the same
    platform, but often you're sacrificing the needs of a particular
    user-group, or future flexibility by not making your system adequately
  2. The hard part of data integration should be contextual

    ​—knowing how information should work together​ to provide some benefit
    for a bunch of humans—but all too often it is technical. We're hoping that
    by knocking down the technical barriers, program managers will be able to
    create and manage more useful connections between tools. In the past,
    setting up and managing an integration was the domain of the IT team—if
    there was one. I think we can make it less "scary".
  3. *Consider letting other parts of your system "write" your ODK forms
    for you. *This may not be a novel idea to many of you, but if you're
    integrating ODK with another system, you may find that it's easier to have
    that destination system generate xForms via a script. If new forms need to
    be generated and disseminated based on changes to objects and fields in
    Salesforce, or data-elements and data-sets in DHIS2, then why not abstract
    this process as it will undoubtedly continue. We've even played around with
    this is a way to generate class-specific xForms with Present/Absent
    questions for each registered participant. Lots of directions you can go
    from here.
  4. Make use of third-party visualization and communication tools. There
    has been an explosion of great mapping, visualization, analytics and even
    notification tools. They're all useless without solid source data, and ODK
    has, time and time again, proven to be invaluable for us as a way to gather
    rich data from mobile phones in disconnected areas. Thinking of ODK as a
    source in live, multi-faceted systems, rather than an "end-to-end"
    deployment suitable for periodic research and data gathering efforts has
    greatly expanded the range of fit across our clients.

​ particularly eager to hear what other technologies have proved valuable
to connect to ODK, but am also happy to dig in on any of the above points.

Also, if anyone is interested in OpenFn (the integration platform we're
building now) please check us out on Github https://github.com/OpenFn/.
Below, I've pasted a sample "expression" that's used to map submissions
from a deeply nested form to multiple parent and child objects in
Salesforce just to get you excited! Users will create those expressions
using a little point-and-click builder https://www.openfn.org/v2. The
whole thing is Elixir and Javascript, using React for the front-end. What
you see below takes a "credentials.json" file and published data from ODK's
JSON publisher. It makes use of a 'language-pack' for Salesforce. Anyway, I
hope you get a kick out of it, and Happy Holidays to all!

steps( each( join("$.data.data[]", "$.data.formVersion", "formVersion"),
combine( create("Parent_Object__c", fields( field("ODK_formVersion__c",
sourceValue("$.data.formVersion")), field("ODK_time_end__c", sourceValue("
$.data.end")), field("weather__c", sourceValue("$.data.weather")), field("
ODK_device__c", sourceValue("$.data.deviceid")), field("lat__c", sourceValue
("$.data.geopoint:Latitude")), field("lon__c", sourceValue("
$.data.geopoint:Longitude")), field("ODK_uuid__c", sourceValue("
$.data.meta-instance-id")), field("landing_site__c", sourceValue("
$.data.landing_site")) )), each( join("$.data.rpt_boats[
]", "
$.references[0].id", "parentId"), combine( create("Boat__c", fields( field("
parent__c", sourceValue("$.data.parentId")), field("reg_type__c",
sourceValue("$.data.reg_type")), field("comment_image_url_long__c",
sourceValue("$.data.comment_image")), field("engine_capacity__c",
sourceValue("$.data.engine_capacity")), field("catch_specie_list__c",
function(state) { return Array.apply( null, sourceValue("
$.data.catch_specie_list")(state) ).join(', ') }) )), each( join("
$.data.rpt_catch[]", "$.references[0].id", "Id"), combine( create("Catch__c
", fields( field("parent__c", sourceValue("$.data.Id")), lookup("
classification_record__r", "name", "$.data.selected_specie"), field("
catch_bait_manual__c", sourceValue("$.data.catch_bait_manual")) )), each(
]", "$.references[0].id", "Id"), create("
CCM_Sample__c", fields( field("parent__c", sourceValue("$.data.Id")), field(
"sample_berry__c", sourceValue("$.data.sample_berry")), )) ) ) ) )) ) ))