ODK Aggregate form upload via commandline (curl)

Development
1

Not a question per se, but I just wanted to post it here for posterity in case anybody else ever wants to try this... You can upload an XForm to ODK Aggregate from the commandline using curl with:

curl -X POST -F form_def_file=@helloworld.xml https://123.456.78.90:8443/ODKAggregate/formUpload

The necessary API is ostensibly documented in the Briefcase Aggregate API, but its not clear how to actually translate it to more 'raw' HTTP POST tools, like curl...

BTW I noted it is not necessary to specify the formId anywhere, instead it get picks out of the XForm's <data id="helloworld"> XML element [if you dont get the form_def_file= bit right you'll get back an error saying "Did not receive Form Name and Form XML description", but obviously the name is not required to be passed explicitly in an HTTP param].

1 Like
2

Other useful curl commands...

Get a form list
curl --header "X-OpenRosa-Version: 1.0" https://example.appspot.com/formList

Submit a form
curl -v -H "Content-Type: multipart/form-data" -F "xml_submission_file=@/tmp/submission.xml" https://example.appspot.com/submission

Approaches to stress testing ODK servers also has some useful tools if you want to post lots of submissions.

3

Actually, I've found the header isnt required; this works fine too

Get form list
curl https://123.456.78.90:8443/ODKAggregate/xformsList

and for submitting a form with photos I've been doing this (although the Date header is probably optional)...

Submit a form
curl -X POST -F "xml_submission_file=@instance.xml;type=text/xml" -F"pic=@picture.jpg" -H "X-OpenRosa-Version:1.0" -H "Date:Thu, 09 Mar 2017 14:03:47 NZDT" https://123.456.78.90:8443/ODKAggregate/submission

FYI I find these are commandlines are useful when doing initial post-install tests after setting up a new ODK Aggregate host, to make sure everything working.

4

I think skipping the header and hitting xformsList is not standards compliant. It'll work with Aggregate for now, but probably not other servers.

5

@Xiphware is there detailed doc about manipulating openrosa by "curl" like this?

6

Not really. But the trickiest ones to get right in curl are the two http POST methods - (1) for uploading a new form and, even more so, (2) submitting a multipart form submission - as described above. The rest of the APIs are http GETs, which are fairly straight-forward to translate into curl. These include:

List of forms:

curl -v -X GET https://opendatakit.appspot.com/formList

which returns (emphasis added):

< HTTP/2 200
< content-type: text/xml;charset=utf-8
< x-cloud-trace-context: 10947aa770a888cf0ffd142986c6707f;o=1
< date: Mon, 14 Jan 2019 20:10:47 GMT
< server: Google Frontend
< content-length: 106
< alt-svc: quic=":443"; ma=2592000; v="44,43,39,35"

<forms>
<form url="https://opendatakit.appspot.com/formXml?formId=all-widgets">All widgets</form>
</forms>

Get a form (using URL obtained from formList):

curl -v -X GET 'https://opendatakit.appspot.com/formXml?formId=all-widgets'

which returns:

< HTTP/2 200
< content-type: text/xml;charset=utf-8
< content-disposition: attachment; filename="All_widgets.xml";
< x-cloud-trace-context: e4a864e7cff083c74b98136b648dcad1;o=1
< date: Mon, 14 Jan 2019 20:12:13 GMT
< server: Google Frontend
< content-length: 51504
< alt-svc: quic=":443"; ma=2592000; v="44,43,39,35"

<?xml version="1.0"?>
<h:html xmlns="http://www.w3.org/2002/xforms" xmlns:ev="http://www.w3.org/2001/xml-events" xmlns:h="http://www.w3.org/1999/xhtml" xmlns:jr="http://openrosa.org/javarosa" xmlns:odk="http://www.opendatakit.org/xforms" xmlns:orx="http://openrosa.org/xforms" xmlns:xsd="http://www.w3.org/2001/XMLSchema">
  <h:head><!-- ODK Aggregate upload time: 2018-08-17T16:07:57.243+0000 on https://opendatakit.appspot.com -->
    <h:title>All widgets</h:title>
...
  </h:body>
</h:html>

Get form manifest/attachments

curl -v -X GET 'https://opendatakit.appspot.com/xformsManifest?formId=all-widgets'

which returns:

< HTTP/2 200
< x-openrosa-version: 1.0
< x-openrosa-accept-content-length: 10485760
< content-type: text/xml;charset=utf-8
< x-cloud-trace-context: 83abba5b008de6f65aaa53b00069f0a5
< date: Mon, 14 Jan 2019 20:17:14 GMT
< server: Google Frontend
< content-length: 2210
< alt-svc: quic=":443"; ma=2592000; v="44,43,39,35"

<manifest xmlns="http://openrosa.org/xforms/xformsManifest">
<mediaFile><filename>d.jpg</filename><hash>md5:42ce233d595e0b471f68bbadf3a96c74</hash>
<downloadUrl>https://opendatakit.appspot.com/xformsDownload?blobKey=aggregate.opendatakit.org%3AFormInfo%5B%40version%3D1+and+%40uiVersion%3D0%5D%2F_form_info%5B%40key%3Dmd5%3Acdb4409d837e35990f1cca789e61eb62%5D%2F_form_info_fileset%5B%40ordinal%3D1%5D%2FmanifestFileset%5B%40ordinal%3D1%5D&amp;as_attachment=true</downloadUrl></mediaFile>
...
</manifest>

Get list of submissions for a form (chunked):

curl -v -X GET 'https://opendatakit.appspot.com/view/submissionList?formId=all-widgets'

< HTTP/2 200
< x-openrosa-version: 1.0
< x-openrosa-accept-content-length: 10485760
< content-type: text/xml;charset=utf-8
< x-cloud-trace-context: e8f00313b0fccf561cc6f268f9173eb0
< date: Mon, 14 Jan 2019 20:19:48 GMT
< server: Google Frontend
< content-length: 68860
< alt-svc: quic=":443"; ma=2592000; v="44,43,39,35"

<idChunk xmlns="http://opendatakit.org/submissions"><idList><id>uuid:6136f5c9-238e-40f5-ae4c-5878b88cc13d</id>
<id>uuid:a2b5c558-1b8d-40e8-adb8-8507ef5e1529</id>
<id>uuid:ebbef628-3b9f-4725-9b2d-210dae91e525</id>
<id>uuid:405a72f3-6dae-4717-b3a6-07eb4feb7f18</id>
...
</idList><resumptionCursor>&lt;cursor xmlns="http://www.opendatakit.org/cursor"&gt;&lt;attributeName&gt;_LAST_UPDATE_DATE&lt;/attributeName&gt;&lt;attributeValue&gt;2019-01-14T17:16:58.168+0000&lt;/attributeValue&gt;&lt;uriLastReturnedValue&gt;uuid:faa3df1d-3185-494c-a69f-1a5b2e39d87c&lt;/uriLastReturnedValue&gt;&lt;isForw

For the most part you can just perform an http GET on the appropriate URL (usually constructed from the data received via other GETs...), check for a 200 OK response, and then parse the resulting XML data.

Edit: oh, as @yanokwa indicates, strictly speaking you should probably pass up the --header "X-OpenRosa-Version: 1.0" header too on all your GETs (although depending on the actual server it may/not always be checked).

1 Like
7

Thanks for these first tips.
sharply more help for POST method.

curl -v -X GET https://opendatakit.appspot.com/formList

Why do not we need authentication to get a list of forms like that?

8

The demo opendatakit.appspot.com site has been setup for anonymous browsing. Typically you'd setup specific user permissions, in which case you'd have to pass up appropriate http authorization headers too. See also https://docs.opendatakit.org/openrosa-authentication/ for Aggregate authentication (or https://odkcentral.docs.apiary.io/#reference/authentication for Central).

1 Like