Custom endpoints & auth for aggregate replacement

Hi All! I'm in the process of building a simple "Aggregate replacement" API / server in NodeJS to

  • receive ODK Collect form submissions, and
  • replicate the endpoints used by ODK Collect as closely as possible.

I'm struggling to find the endpoint Collect uses to download the custom media that accompanies a form. In Collect > Server settings > Type = Other, there are only two options for endpoints - "Form list path" and "Submission Path". The form list contains only direct links to form definitions. My questions are:
Q - how does aggregate expose the media files required by a form to Collect for download?
Q - how does collect authenticate with aggregate, if one was to replicate this? (basic / digest / other? )

The information is documented if you search for it. You are going to be interested in the Collect-Aggregate API.

http://www.giyf.com/odk%20aggregate%20api

A - Media manifests through form list
A - http digest and basic auth over TLS

Hi Brent, thank you for your response.

I may have been slightly vague in my first post - I've been looking through the docs carefully - the only mention of the way "other media" is downloaded with a form definition by Collect is on the Javarosa FormListAPI wiki page. The API docs show an example form list response as follows:

<xforms xmlns="http://openrosa.org/xforms/xformsList">
  <xform>
    <formID>mydomain.org:formId</id>
    <name>Form with zero or more additional files</name>
    <version>1.1</version>
    <hash>md5:c28fc778a9291672badee04ac880a05d</hash>
    <descriptionText>A possibly very long description of the form</descriptionText>
    <downloadUrl>http://myhost.com/app/path/getMe/formIdA</downloadUrl>
    <manifestUrl>http://myothehost.com/app/path/getOtherStuff?formId=formIdA</manifestUrl>
  </xform>
...
</xforms>

This makes sense - there is a manifestUrl tag which points to the extra files. However, the xml response by the /formList endpoint on odk aggregate does not follow this specification. Instead I'm getting:

<forms>
    <form url="https://abalobi-fisher.appspot.com/formXml?formId=Fisher_Logbook_v2_0">Fisher_Logbook_v2_0</form>
</forms>

From what I can see, the only reference to this specfic functionality on the github wiki page is the Download forms and media files section on the Briefcase Aggregate API page.

I'm asking here on the forum short of having to delve into the ODK Collect source myself.

First, don't be shy about looking at the source code. It's the only 100% reliable source of truth when it comes to these types of questions. However, having reverse engineered the protocol for Aggregate and Briefcase myself recently, I can say the docs are fairly accurate.

One reason you may be seeing differences is that if you try to use a normal web browser to inspect the protocol, you're not going to observe what a compliant OpenRosa client would. You need to send the OpenRosa headers for Aggregate to speak the full protocol. You're likely seeing the fallback compatibility mode for pre-OpenRosa clients.

HTH,

Brent

I believe that what you're looking for is at https://bitbucket.org/javarosa/javarosa/wiki/FormListAPI. There's a project underway to verify that documentation and make it available at https://github.com/opendatakit/docs so hopefully it will soon be easier to find.

The Briefcase API is another animal that you probably won't need to touch unless you also want Briefcase to be able to pull from your server. Collect does not interact with it directly.

edit: now I see that it's the doc you had linked to in the first place. That's the one to look at and @Batkison's explanation of why you're not seeing the expected response sounds right.

You might be interested in taking a look at OpenMapKitServer. It's the lightweight NodeJS server component of OpenMapKit that handles the collection and aggregation of OpenStreetMap and OpenDataKit data. But it works even if you don't have an OMK part to your survey.

2 Likes

Hi everyone, thanks for the replies. For anyone who sees this thread in the future, I needed to set the OpenRosa header as Brent suggested.

"X-OpenRosa-Version" : "1.0"

Thanks for all the help! @danbjoseph I'll definitely take a look at the odk portion of OpenMapKitServer, this is exactly what I've been looking for.

2 Likes

Where can I access https://bitbucket.org/javarosa/javarosa/wiki/FormListAPI the link is broken.

Hi @tmndungu, welcome to the ODK forum, we're glad you're here! Thanks for adding a picture to your profile, we think it helps build community. If you get a chance, we'd appreciate it if you introduce yourself on this other forum thread.

I believe those docs have been transitioned to docs.getodk.org and can be found at
https://docs.getodk.org/openrosa/#openrosa-1-0-apis (and with the Form List API docs at https://docs.getodk.org/openrosa-form-list/)

1 Like

Hi @danbjoseph, thank you for the warm welcome and quick response to my query. I have put a brief intro hope its sufficient.

1 Like