How to retrieve data from ODK-Collect while offline

I am trying to retrieve form data without using briefcase/aggregate since I will be strictly in an offline environment. I want to extract the data after a user has filled out a form and parse it for my own purposes in a separate application. Is this possible through some sort of Intent?

I have a working concept but wondering if there is a better/officially supported workflow. I could be off on how to go about this as I am just getting familiar with ODK and its capabilities.

Problem:
Extract data from ODK Collect form after user has filled out the data. This must occur in an offline environment.

Workflow:
User opens external app and through workflow is directed to ODK Collect through Intents. User collects data using ODK Collect. User finishes collecting data and is returned to external app. The external app then consumes the data the user just filled out.

Questions
Is extracting/removing data from a filled out ODK collect form via an Intent a supported workflow? If not is there any reason why the below working concept would not work.

The result from:
Intent intent = new Intent(Intent.ACTION_EDIT);
intent.setData(formUri);
startActivityForResult(intent,EDIT_FORM_REQUEST);
returns something like this: content://org.odk.collect.android.provider.odk.instances/instances/6
However there does not seem to be documentation on what to do with this. Is there another intent we can serve this to, like perhaps to get the data from it?

Working concept:

  1. Launch ODK form list activity from app
    Intent intent = new Intent(Intent.ACTION_PICK);
    intent.setType("vnd.android.cursor.dir/vnd.odk.form");
    startActivityForResult(intent, PICK_FORM_REQUEST);

  2. Recieve restult from app and direct to the form
    @Override
    public void onActivityResult(int requestCode, int resultCode, Intent data) {
    if (requestCode == PICK_FORM_REQUEST) {
    if (resultCode == RESULT_OK) {
    Uri formUri = data.getData();
    Intent intent = new Intent(Intent.ACTION_EDIT);
    intent.setData(formUri);
    startActivityForResult(intent,EDIT_FORM_REQUEST);
    }
    }
    }

  3. Consume data after form is filled out. Solution below sort of works but feels very hackish. Basically I get the .xml form stored in the instances path. My plan is to then manually parse the .xml data
    public static final String ODK_ROOT = Environment.getExternalStorageDirectory()
    + File.separator + "odk";
    public static final String INSTANCES_PATH = ODK_ROOT + File.separator + "instances";

    @Override
    public void onActivityResult(int requestCode, int resultCode, Intent data) {
    if (requestCode == EDIT_FORM_REQUEST) {
    // Consume data here.
    // This is where it gets messy and doesn't seem like a workflow if supported
    // Current solution below
    String string = data.getDataString();
    // ex: string = content://org.odk.collect.android.provider.odk.instances/instances/6
    try {
    File instancesDirectory = new File(INSTANCES_PATH);
    if (instancesDirectory.exists()) {
    File formsFilledOutDirectories = instancesDirectory.listFiles();
    File lastFormfilledOutDirectory = formsFilledOutDirectories[formsFilledOutDirectories.length - 1];
    File formOfInterestDir = lastFormfilledOutDirectory.listFiles();
    File ActualFormFile = formOfInterestDir[formOfInterestDir.length-1];
    if (ActualFormFile.exists()) {
    InputStream in = new FileInputStream(new File(ActualFormFile.toURI()));
    Document doc = DocumentBuilderFactory.newInstance().newDocumentBuilder().parse(in);
    Element root = doc.getDocumentElement(); // perform DOM operations starting here.
    }
    }
    }catch (Exception e){
    e.printStackTrace();
    }
    }
    }

Thanks for any input,

Welcome to the community, @fkaye!

Can you tell us a little bit about the high-level problem you are trying to solve? There might be a different approach than grabbing individual values from the XML.

As to your proposed solution, you are sort of doing it the way I'd do it, but I tend to do things in a hacky way :sob: Is there an example of other Android apps that do this sort of thing? If yes, what do those apps do? If no, what you expect the Intent call to look like?

Thanks for the quick reply! I really love how active this community is.

So a bit more about the problem I'm trying to solve:

I have an existing mapping application that handles detailed geometry capture. It also handles attribute data collection for the the geometric features that they are collecting. However, my app only supports generic attribute capture. I want to be able to include a more customizable attribute data collection workflow. After some research I decided to work with the XForm standard, thus ODK-Collect. Basically In my app I am wanting my app to be able to render forms based on XForms.

Since in my application I have an existing database -- I only need the form rendering and a return of the data.

At first I thought I would just wrap ODK into my app, turn it into a Library. After some trial and error I figured this would be much more work that it was worth and would likely break things along the way. Also I am hesitant to just import the FormEntryActivity related code and embed it into the application, as this seems like a lot of work also. Then I discovered the Intents methods.

Am I correct in that there is not an easy way to wrap collect into an existing app? There is no API or library version collect right?

Again this must all occur offline as my users will be out range of connectivity almost always. This seems relevant as some proposed solutions might involve using the existing functionality in ODK collect to push data up to servers and harvest it dynamically from my app.

This thread also describes what I'm facing almost exactly: Integrating ODK Collect

Thanks again,

Thanks for the additional detail! You are correct that there isn't an easy way to wrap Collect. The best you can do is to use the intents. As an aside, @Grzesiek2010's Collect Intents Tester app is great for this.

I think the current setup that we have where you get back a URI to the instance feels good enough to me. Maybe we can make that nicer by adding an extra where you get back the path to the raw XML. Anything else (e.g., you pass in a path to a node and we return that value) feels decadent.

I'd be curious what others (especially @Kigamba who has been extending the intents in Collect) think.

Hi @fkaye,

Welcome to the community,
First, sorry for the late reply

  1. You can check if the ContentProvider used by ODK Collect will help. I mostly know that it stores basic information
  2. Any data sent to the server is in the same form of XML
  3. Parsing the XML which might be the simplest form of the data(or JSON) without breaking current functionality/features, seems like the best idea. Any other simpler form needs you to think about handling complex question types and their answers - ODK has tried to create SMS submissions -> This might be the easiest form of the data which is still not complete

I am not sure if the little above helped.

The above is what I can provide with the use-case you provided.

I have not checked the form in which the resultant data is in. @fkaye Probably what's the structure in which you would want the data?

I think it would be a good feature to have. I am just not sure on the path to take.

Having it in ODK Collect kind of ensures any changes that affect data and structure of forms is maintained from a central point which is ODK Collect and you do not have to manage duplicate code on the same

Ok great. Thanks for the response @Kigamba. Good to know that I'm not off the mark on how to tackle this.

Yes it would be cool to come up with a way to incorporate some feature to better support this workflow. I'd want to be able to support related data workflow. I haven't dug into what that looks like in the .xml but I'd hope that it isn't too difficult to parse.

Its just structured (ie potentially hierarchical groupings) of XML. This is an example of what gets submitted:

<?xml version="1.0" encoding="utf-8"?>
<data id="snapshot_xml">
  <name>Gareth</name>
  <job_title/>
  <group_tf6oj91>
    <reg_1>yes</reg_1>
    <reg_2>yes</reg_2>
  </group_tf6oj91>
  <group_la1wt43>
    <mort_1>1</mort_1>
    <mort_1_description/>
    <mort_1_photo/>
    <mort_2>1</mort_2>
    <mort_2_description/>
    <mort_2_photo/>
    <mort_3>1</mort_3>
    <mort_3_description/>
    <mort_3_photo/>
    <mort_4>1</mort_4>
    <mort_4_description/>
    <mort_4_photo/>
    <mort_5>1</mort_5>
    <mort_5_description/>
    <mort_5_photo/>
  </group_la1wt43>
<signature>DA0EBA25-6C50-4872-BDF3-0CF8C549B02A.jpg</signature>
  <signature_of_on_site_represent>B40F100C-ECE4-4CBE-8EE0-A09D75FF4E74.jpg</signature_of_on_site_represent>
  <meta>
    <instanceID/>
  </meta>
</data>

There are a number of freely available tools to convert this (or any) XML into different formats, eg if you put the above thru xml2json you get this:

{
  "data": {
    "-id": "snapshot_xml",
    "name": "Gareth",
    "group_tf6oj91": {
      "reg_1": "yes",
      "reg_2": "yes"
    },
    "group_la1wt43": {
      "mort_1": "1",
      "mort_2": "1",
      "mort_3": "1",
      "mort_4": "1",
      "mort_5": "1"
    },
    "signature": "DA0EBA25-6C50-4872-BDF3-0CF8C549B02A.jpg",
    "signature_of_on_site_represent": "B40F100C-ECE4-4CBE-8EE0-A09D75FF4E74.jpg",
    "meta": {
      
    }
  }
}

or thru xml-to-csv

data/name,data/job_title,data/group_tf6oj91/reg_1,data/group_tf6oj91/reg_2,data/group_la1wt43/mort_1,data/group_la1wt43/mort_1_description,data/group_la1wt43/mort_1_photo,data/group_la1wt43/mort_2,data/group_la1wt43/mort_2_description,data/group_la1wt43/mort_2_photo,data/group_la1wt43/mort_3,data/group_la1wt43/mort_3_description,data/group_la1wt43/mort_3_photo,data/group_la1wt43/mort_4,data/group_la1wt43/mort_4_description,data/group_la1wt43/mort_4_photo,data/group_la1wt43/mort_5,data/group_la1wt43/mort_5_description,data/group_la1wt43/mort_5_photo,data/signature,data/signature_of_on_site_represent,data/meta/instanceID,data/_id
Gareth,,yes,yes,1,,,1,,,1,,,1,,,1,,,DA0EBA25-6C50-4872-BDF3-0CF8C549B02A.jpg,B40F100C-ECE4-4CBE-8EE0-A09D75FF4E74.jpg,,snapshot_xml

et cetera, et cetera...

Or, if you really want to actually parse the XML yourself, there are similarly a number of open source tools at your disposal; firstly you'll probably have to decide whether to go with a DOM vs SAX approach.