Upload and download APIs

Support
#1

Hi all,

We're trying to programmatically upload completed xforms to the Aggregate
server. (We need to develop and test the data visualisation, and we have
some code to generate xforms submissions filled with random values to do
this).

We're trying to figure out how to do this. The protocol between the client
and server seems complex and arbitrary. I notice that the ODK Collect
InstanceUploaderTask contains the comment: "TODO: This method is like 350
lines long, down from 400. still. ridiculous. make it smaller."

Is there another, simpler or more standard API that we can use to upload
instance data? Is there any API to download data from Aggregate?

We've been trying to log into the Aggregate server (hosted on AppEngine)
using Basic and Digest authentication to the application root URL with
Curl. But whatever we do, we just get sent to the OpenID login page. It's
not clear to me exactly how this is different to what the phone is doing?
Is it the OpenRosa HTTP headers that make the difference?

curl -X HEAD -u htc:htc --basic https://partimob.appspot.com/submission -v

  • About to connect() to partimob.appspot.com port 443 (#0)
  • Trying 74.125.79.141... connected
  • Connected to partimob.appspot.com (74.125.79.141) port 443 (#0)
  • successfully set certificate verify locations:
  • CAfile: none
    CApath: /etc/ssl/certs
  • SSLv3...
  • Server auth using Basic with user 'htc'

HEAD /submission HTTP/1.1
Authorization: Basic aHRjOmh0Yw==
User-Agent: curl/7.21.6 (i686-pc-linux-gnu) libcurl/7.21.6
OpenSSL/1.0.0e zlib/1.2.3.4 libidn/1.22 librtmp/2.3
Host: partimob.appspot.com
Accept: /

< HTTP/1.1 302 Found
< Expires: Thu, 01 Jan 1970 00:00:00 GMT
< Set-Cookie: JSESSIONID=lc5G3yIFRQZKRmMiJKyTLQ;Path=/
< Location:
https://partimob.appspot.com/openid_login.html;jsessionid=lc5G3yIFRQZKRmMiJKyTLQ
< Date: Wed, 09 Nov 2011 15:30:24 GMT
< Content-Type: text/html
< Server: Google Frontend
< Content-Length: 0
<

  • Connection #0 to host partimob.appspot.com left intact
  • Closing connection #0
  • SSLv3, TLS alert, Client hello (1):

Cheers, Chris.

··· -- Aptivate | http://www.aptivate.org | Phone: +44 1223 760887 The Humanitarian Centre, Fenner's, Gresham Road, Cambridge CB1 2ES

Aptivate is a not-for-profit company registered in England and Wales
with company number 04980791.

#2

If you are using Aggregate 1.0, consider the ODK Briefcase application. It
can bulk-upload submissions to Aggregate.

It was written to do exactly what you're trying to do and is what I use to
load up test data for my own testing.

Mitch

··· On Wed, Nov 9, 2011 at 7:32 AM, Chris Wilson wrote:

Hi all,

We're trying to programmatically upload completed xforms to the Aggregate
server. (We need to develop and test the data visualisation, and we have
some code to generate xforms submissions filled with random values to do
this).

We're trying to figure out how to do this. The protocol between the client
and server seems complex and arbitrary. I notice that the ODK Collect
InstanceUploaderTask contains the comment: "TODO: This method is like 350
lines long, down from 400. still. ridiculous. make it smaller."

Is there another, simpler or more standard API that we can use to upload
instance data? Is there any API to download data from Aggregate?

We've been trying to log into the Aggregate server (hosted on AppEngine)
using Basic and Digest authentication to the application root URL with
Curl. But whatever we do, we just get sent to the OpenID login page. It's
not clear to me exactly how this is different to what the phone is doing?
Is it the OpenRosa HTTP headers that make the difference?

curl -X HEAD -u htc:htc --basic https://partimob.appspot.com/**submissionhttps://partimob.appspot.com/submission-v

  • About to connect() to partimob.appspot.com port 443 (#0)
  • Trying 74.125.79.141... connected
  • Connected to partimob.appspot.com (74.125.79.141) port 443 (#0)
  • successfully set certificate verify locations:
  • CAfile: none
    CApath: /etc/ssl/certs
  • SSLv3...
  • Server auth using Basic with user 'htc'

HEAD /submission HTTP/1.1
Authorization: Basic aHRjOmh0Yw==
User-Agent: curl/7.21.6 (i686-pc-linux-gnu) libcurl/7.21.6

OpenSSL/1.0.0e zlib/1.2.3.4 libidn/1.22 librtmp/2.3

Host: partimob.appspot.com
Accept: /

< HTTP/1.1 302 Found
< Expires: Thu, 01 Jan 1970 00:00:00 GMT
< Set-Cookie: JSESSIONID=**lc5G3yIFRQZKRmMiJKyTLQ;Path=/
< Location: https://partimob.appspot.com/openid_login.html;jsessionid=
lc5G3yIFRQZKRmMiJKyTLQhttps://partimob.appspot.com/openid_login.html;jsessionid=lc5G3yIFRQZKRmMiJKyTLQ
< Date: Wed, 09 Nov 2011 15:30:24 GMT
< Content-Type: text/html
< Server: Google Frontend
< Content-Length: 0
<

  • Connection #0 to host partimob.appspot.com left intact
  • Closing connection #0
  • SSLv3, TLS alert, Client hello (1):

Cheers, Chris.

Aptivate | http://www.aptivate.org | Phone: +44 1223 760887
The Humanitarian Centre, Fenner's, Gresham Road, Cambridge CB1 2ES

Aptivate is a not-for-profit company registered in England and Wales
with company number 04980791.

--
Post: opendatakit@googlegroups.com
Unsubscribe: opendatakit+unsubscribe@**googlegroups.comopendatakit%2Bunsubscribe@googlegroups.com
Options: http://groups.google.com/**group/opendatakit?hl=enhttp://groups.google.com/group/opendatakit?hl=en

--
Mitch Sundt
Software Engineer


University of Washington
mitchellsundt@gmail.com
#3

Hi Mitch and all,

··· On Wed, 9 Nov 2011, Mitch Sundt wrote:

If you are using Aggregate 1.0, consider the ODK Briefcase application.
It can bulk-upload submissions to Aggregate.

Thanks for that suggestion. I hadn't considered using it for this purpose
but it makes sense. I will investigate the APIs that it uses to upload and
download data. I'm currently running training for a live deployment in
Zambia that will be starting next week.

I was hoping to use the integration with Google Fusion or some other
online system to generate live graphs of data and build workflows around
it. We did modify Fusion to allow specifying the server parameters on the
command line, to make it easier for our users, and you can find that
modified version at:
http://code.google.com/r/tomd-briefcase/source/browse.

We managed to get the upload using cURL working fairly reliably by adding
headers and a short delay between requests, and we have a Python script to
generate and upload random form submissions.

Now we're working on fixing the Google Fusion Tables integration. It seems
that Aggregate integration doesn't work properly if one uses a GAFYD
(custom domain) login, generating the error message that I reported
before. It does work if one logs into Google using a gmail address.

However the streaming of new uploads to Fusion Tables doesn't seem to work
properly. It does upload one or two submissions, but then stops working.
The application logs show the upload task appearing to succeed, but the
data never makes it into Fusion Tables. Has anyone seen that behaviour?

Also, I though ODK Collect was supposed to save completed instances on the
SD card, and indeed it seems to fail to save the form (after it's been
completed!) unless an SD card is installed in the device, but I can't
actually find the files on the SD card, only on the device's internal
storage. Does anyone know more about that?

Cheers, Chris.

Aptivate | http://www.aptivate.org | Phone: +44 1223 760887
The Humanitarian Centre, Fenner's, Gresham Road, Cambridge CB1 2ES

Aptivate is a not-for-profit company registered in England and Wales
with company number 04980791.

#4

For GAFYD logins, in another context, I found that my GAFYD didn't work
with Google Apps until I "created" a Google Apps login that matched my
GAFYD login, and then it appeared to work (!?) for that context.

Basically, in my case, I had a GAFYD, but it didn't have a Google Apps
account, so I couldn't see any Docs (Google Documents) using that login.
Once I "created" a Google Apps account, and I could navigate to and access
the Docs page, the app I was trying to use started working properly. It
may be that you have a similar situation?

In Aggregate 1.0 and earlier, there was a bug that didn't flush the most
recently submitted data to Fusion Tables or Google Spreadsheet. Aggregate
1.0.1 fixed that. If you haven't already, upgrade to 1.0.1 and see if the
publishing issue still exists. If it does, please let me know or open an
issue for it.

ODK Collect saves the instance data in directories under
/sdcard/odk/instances

I believe the download interfaces are stable, but keep in mind that they
are considered internal and may change going forward - particularly if it
makes the Briefcase interactions more efficient.

Mitch

··· On Mon, Nov 14, 2011 at 8:18 AM, Chris Wilson wrote:

Hi Mitch and all,

On Wed, 9 Nov 2011, Mitch Sundt wrote:

If you are using Aggregate 1.0, consider the ODK Briefcase application.

It can bulk-upload submissions to Aggregate.

Thanks for that suggestion. I hadn't considered using it for this purpose
but it makes sense. I will investigate the APIs that it uses to upload and
download data. I'm currently running training for a live deployment in
Zambia that will be starting next week.

I was hoping to use the integration with Google Fusion or some other
online system to generate live graphs of data and build workflows around
it. We did modify Fusion to allow specifying the server parameters on the
command line, to make it easier for our users, and you can find that
modified version at:
<http://code.google.com/r/**tomd-briefcase/source/browsehttp://code.google.com/r/tomd-briefcase/source/browse

.

We managed to get the upload using cURL working fairly reliably by adding
headers and a short delay between requests, and we have a Python script to
generate and upload random form submissions.

Now we're working on fixing the Google Fusion Tables integration. It seems
that Aggregate integration doesn't work properly if one uses a GAFYD
(custom domain) login, generating the error message that I reported before.
It does work if one logs into Google using a gmail address.

However the streaming of new uploads to Fusion Tables doesn't seem to work
properly. It does upload one or two submissions, but then stops working.
The application logs show the upload task appearing to succeed, but the
data never makes it into Fusion Tables. Has anyone seen that behaviour?

Also, I though ODK Collect was supposed to save completed instances on the
SD card, and indeed it seems to fail to save the form (after it's been
completed!) unless an SD card is installed in the device, but I can't
actually find the files on the SD card, only on the device's internal
storage. Does anyone know more about that?

Cheers, Chris.

Aptivate | http://www.aptivate.org | Phone: +44 1223 760887
The Humanitarian Centre, Fenner's, Gresham Road, Cambridge CB1 2ES

Aptivate is a not-for-profit company registered in England and Wales
with company number 04980791.

--
Post: opendatakit@googlegroups.com
Unsubscribe: opendatakit+unsubscribe@**googlegroups.comopendatakit%2Bunsubscribe@googlegroups.com
Options: http://groups.google.com/**group/opendatakit?hl=enhttp://groups.google.com/group/opendatakit?hl=en

--
Mitch Sundt
Software Engineer


University of Washington
mitchellsundt@gmail.com
#5

Hi Mitch,

For GAFYD logins, in another context, I found that my GAFYD didn't work
with Google Apps until I "created" a Google Apps login that matched my
GAFYD login, and then it appeared to work (!?) for that context.

Basically, in my case, I had a GAFYD, but it didn't have a Google Apps
account, so I couldn't see any Docs (Google Documents) using that
login. Once I "created" a Google Apps account, and I could navigate to
and access the Docs page, the app I was trying to use started working
properly. It may be that you have a similar situation?

I'm not sure if I understand correctly the difference between GAFYD and
Google Apps accounts, or how I could have an account for one but not the
other? If my account is example@aptivate.org, on which sites would this be
considered a GAFYD account, and on which would it be a Google Apps
account? Didn't they solve this kind of issue with the whole google
account deduplication pain they put us through this summer?

In Aggregate 1.0 and earlier, there was a bug that didn't flush the most
recently submitted data to Fusion Tables or Google Spreadsheet.
Aggregate 1.0.1 fixed that. If you haven't already, upgrade to 1.0.1
and see if the publishing issue still exists. If it does, please let me
know or open an issue for it.

Thanks for the heads up, will give it a try and let you know.

ODK Collect saves the instance data in directories under /sdcard/odk/instances

Are you 100% sure about that? On the devices I've been using, there's
nothing on the SD card, and all the instances are in /odk/instances.

I believe the download interfaces are stable, but keep in mind that they
are considered internal and may change going forward - particularly if
it makes the Briefcase interactions more efficient.

That's OK, we'll probably stick with a fixed, tested combination of
Collect, Aggregate and Briefcase.

Cheers, Chris.

··· On Mon, 14 Nov 2011, Mitch Sundt wrote: -- Aptivate | http://www.aptivate.org | Phone: +44 1223 760887 The Humanitarian Centre, Fenner's, Gresham Road, Cambridge CB1 2ES

Aptivate is a not-for-profit company registered in England and Wales
with company number 04980791.

#6

Hello,

What headers did you have to add? We are having the same problem.

Paul.

#7

I was also puzzled about the GAFYD issue I saw. All I can say is, that
when I visited Google Apps page and tried to sign in with what I thought
was the converged GAFYD e-mail account, Google Apps rejected me, but then
allowed me to register that same e-mail account. After that, things began
to work. Confusing. Very confusing.

You may have an older Collect -- I'm talking Collect 1.1.5 and higher -- or
you may be using something other than ODK Collect.

Mitch

··· On Mon, Nov 14, 2011 at 11:17 AM, Chris Wilson wrote:

Hi Mitch,

On Mon, 14 Nov 2011, Mitch Sundt wrote:

For GAFYD logins, in another context, I found that my GAFYD didn't work

with Google Apps until I "created" a Google Apps login that matched my
GAFYD login, and then it appeared to work (!?) for that context.

Basically, in my case, I had a GAFYD, but it didn't have a Google Apps
account, so I couldn't see any Docs (Google Documents) using that login.
Once I "created" a Google Apps account, and I could navigate to and access
the Docs page, the app I was trying to use started working properly. It
may be that you have a similar situation?

I'm not sure if I understand correctly the difference between GAFYD and
Google Apps accounts, or how I could have an account for one but not the
other? If my account is example@aptivate.org, on which sites would this
be considered a GAFYD account, and on which would it be a Google Apps
account? Didn't they solve this kind of issue with the whole google account
deduplication pain they put us through this summer?

In Aggregate 1.0 and earlier, there was a bug that didn't flush the most

recently submitted data to Fusion Tables or Google Spreadsheet. Aggregate
1.0.1 fixed that. If you haven't already, upgrade to 1.0.1 and see if the
publishing issue still exists. If it does, please let me know or open an
issue for it.

Thanks for the heads up, will give it a try and let you know.

ODK Collect saves the instance data in directories under

/sdcard/odk/instances

Are you 100% sure about that? On the devices I've been using, there's
nothing on the SD card, and all the instances are in /odk/instances.

I believe the download interfaces are stable, but keep in mind that they

are considered internal and may change going forward - particularly if it
makes the Briefcase interactions more efficient.

That's OK, we'll probably stick with a fixed, tested combination of
Collect, Aggregate and Briefcase.

Cheers, Chris.

Aptivate | http://www.aptivate.org | Phone: +44 1223 760887
The Humanitarian Centre, Fenner's, Gresham Road, Cambridge CB1 2ES

Aptivate is a not-for-profit company registered in England and Wales
with company number 04980791.

--
Post: opendatakit@googlegroups.com
Unsubscribe: opendatakit+unsubscribe@**googlegroups.comopendatakit%2Bunsubscribe@googlegroups.com
Options: http://groups.google.com/**group/opendatakit?hl=enhttp://groups.google.com/group/opendatakit?hl=en

--
Mitch Sundt
Software Engineer


University of Washington
mitchellsundt@gmail.com
#8

chris,

let us know what device you are using collect on and what version you
are using. the app should only write instances to the external storage
(usually the sd card)...

yaw

··· On Mon, Nov 14, 2011 at 11:24, Mitch Sundt wrote: > I was also puzzled about the GAFYD issue I saw. All I can say is, that when > I visited Google Apps page and tried to sign in with what I thought was the > converged GAFYD e-mail account, Google Apps rejected me, but then allowed me > to register that same e-mail account. After that, things began to work. > Confusing. Very confusing. > > You may have an older Collect -- I'm talking Collect 1.1.5 and higher -- or > you may be using something other than ODK Collect. > > Mitch > > On Mon, Nov 14, 2011 at 11:17 AM, Chris Wilson wrote: >> >> Hi Mitch, >> >> On Mon, 14 Nov 2011, Mitch Sundt wrote: >> >>> For GAFYD logins, in another context, I found that my GAFYD didn't work >>> with Google Apps until I "created" a Google Apps login that matched my GAFYD >>> login, and then it appeared to work (!?) for that context. >>> >>> Basically, in my case, I had a GAFYD, but it didn't have a Google Apps >>> account, so I couldn't see any Docs (Google Documents) using that login. >>> Once I "created" a Google Apps account, and I could navigate to and access >>> the Docs page, the app I was trying to use started working properly. It may >>> be that you have a similar situation? >> >> I'm not sure if I understand correctly the difference between GAFYD and >> Google Apps accounts, or how I could have an account for one but not the >> other? If my account is example@aptivate.org, on which sites would this be >> considered a GAFYD account, and on which would it be a Google Apps account? >> Didn't they solve this kind of issue with the whole google account >> deduplication pain they put us through this summer? >> >>> In Aggregate 1.0 and earlier, there was a bug that didn't flush the most >>> recently submitted data to Fusion Tables or Google Spreadsheet. Aggregate >>> 1.0.1 fixed that. If you haven't already, upgrade to 1.0.1 and see if the >>> publishing issue still exists. If it does, please let me know or open an >>> issue for it. >> >> Thanks for the heads up, will give it a try and let you know. >> >>> ODK Collect saves the instance data in directories under >>> /sdcard/odk/instances >> >> Are you 100% sure about that? On the devices I've been using, there's >> nothing on the SD card, and all the instances are in /odk/instances. >> >>> I believe the download interfaces are stable, but keep in mind that they >>> are considered internal and may change going forward - particularly if it >>> makes the Briefcase interactions more efficient. >> >> That's OK, we'll probably stick with a fixed, tested combination of >> Collect, Aggregate and Briefcase. >> >> Cheers, Chris. >> -- >> Aptivate | http://www.aptivate.org | Phone: +44 1223 760887 >> The Humanitarian Centre, Fenner's, Gresham Road, Cambridge CB1 2ES >> >> Aptivate is a not-for-profit company registered in England and Wales >> with company number 04980791. >> >> -- >> Post: opendatakit@googlegroups.com >> Unsubscribe: opendatakit+unsubscribe@googlegroups.com >> Options: http://groups.google.com/group/opendatakit?hl=en > > > > -- > Mitch Sundt > Software Engineer > http://www.OpenDataKit.org > University of Washington > mitchellsundt@gmail.com > > -- > Post: opendatakit@googlegroups.com > Unsubscribe: opendatakit+unsubscribe@googlegroups.com > Options: http://groups.google.com/group/opendatakit?hl=en >
#9

Hi Yaw and all,

··· On Mon, 14 Nov 2011, Yaw Anokwa wrote:

let us know what device you are using collect on and what version you
are using. the app should only write instances to the external storage
(usually the sd card)...

We are using a version that reports itself as 1.1.7. I think it might be
RC2. (Internet is very slow here, so difficult to download again to
confirm). It is not creating any files on the external SD card at all.

I think Android refers to its internal memory as an "internal SD card" in
some places. Could this be a source for confusion? Under this terminology,
the files are on the "internal SD card" in /odk/instances, but when I
remove the "external" SD card and insert it into another machine, they are
not there.

Cheers, Chris.

Aptivate | http://www.aptivate.org | Phone: +44 1223 760887
The Humanitarian Centre, Fenner's, Gresham Road, Cambridge CB1 2ES

Aptivate is a not-for-profit company registered in England and Wales
with company number 04980791.

#10

Hi Mitch and all,

··· On Mon, 14 Nov 2011, Chris Wilson wrote:

In Aggregate 1.0 and earlier, there was a bug that didn't flush the
most recently submitted data to Fusion Tables or Google Spreadsheet.
Aggregate 1.0.1 fixed that. If you haven't already, upgrade to 1.0.1
and see if the publishing issue still exists. If it does, please let
me know or open an issue for it.

Thanks for the heads up, will give it a try and let you know.

I'm afraid it doesn't work. I have a 1.0.1 instance with 38 instances. I
export this to Fusion Tables (even just the existing data, not streaming
new data) and it stops after sending 8 of them.

This is a real shame because we want to build a desktop data review system
(perhaps in Microsoft Access) and ideally I'd like it to be linked live to
an online system, but since Fusion Tables integration is broken, we can't
rely on it, and the only thing I can do is integrate with the unstable
interfaces of Collect itself. Similarly we can't use the easy
visualisation tools provided by Google because we can't get the data into
Fusion Tables.

Is there anything I can do to help debug this in the next couple of days?

Cheers, Chris.

Aptivate | http://www.aptivate.org | Phone: +44 1223 760887
The Humanitarian Centre, Fenner's, Gresham Road, Cambridge CB1 2ES

Aptivate is a not-for-profit company registered in England and Wales
with company number 04980791.

#11

Moving to developer list.

The thread you are responding to is a few years old. See
https://bitbucket.org/javarosa/javarosa/wiki/FormSubmissionAPI for the
API to implement.

What problem are you having with that API?

Yaw

··· -- Need ODK services? http://nafundi.com provides form design, server setup, professional support, and software development for ODK.

On Wed, Sep 10, 2014 at 3:14 PM, paul.gardner.stephen@gmail.com wrote:

Hello,

What headers did you have to add? We are having the same problem.

Paul.

--

Post: opendatakit@googlegroups.com
Unsubscribe: opendatakit+unsubscribe@googlegroups.com
Options: http://groups.google.com/group/opendatakit?hl=en

You received this message because you are subscribed to the Google Groups "ODK Community" group.
To unsubscribe from this group and stop receiving emails from it, send an email to opendatakit+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

#12

What version of Android are you using?

Mitch

··· On Tue, Nov 15, 2011 at 4:25 AM, Chris Wilson wrote:

Hi Yaw and all,

On Mon, 14 Nov 2011, Yaw Anokwa wrote:

let us know what device you are using collect on and what version you are

using. the app should only write instances to the external storage (usually
the sd card)...

We are using a version that reports itself as 1.1.7. I think it might be
RC2. (Internet is very slow here, so difficult to download again to
confirm). It is not creating any files on the external SD card at all.

I think Android refers to its internal memory as an "internal SD card" in
some places. Could this be a source for confusion? Under this terminology,
the files are on the "internal SD card" in /odk/instances, but when I
remove the "external" SD card and insert it into another machine, they are
not there.

Cheers, Chris.

Aptivate | http://www.aptivate.org | Phone: +44 1223 760887
The Humanitarian Centre, Fenner's, Gresham Road, Cambridge CB1 2ES

Aptivate is a not-for-profit company registered in England and Wales
with company number 04980791.

--
Post: opendatakit@googlegroups.com
Unsubscribe: opendatakit+unsubscribe@**googlegroups.comopendatakit%2Bunsubscribe@googlegroups.com
Options: http://groups.google.com/**group/opendatakit?hl=enhttp://groups.google.com/group/opendatakit?hl=en

--
Mitch Sundt
Software Engineer


University of Washington
mitchellsundt@gmail.com
#13

Hi Mitch,

··· On Tue, 15 Nov 2011, Mitch Sundt wrote:

What version of Android are you using?

This is 2.2 (FROYO.XWJJ7) on a Samsung Galaxy Tab GT-P1000, and I've tried
1.1.7-RC2 and trunk (with the new menu structure) and neither of them save
anything on the (removable) SD card.

I noticed when using ADB to install our custom version (with the
constraint checks skipped on save) that it has the option to install an
application to the SD card instead of the device's main memory, and I'm
wondering whether the location of the application might affect where the
data is saved?

Thank you all for your help by the way! It's much appreciated. I hope to
give something back to the community.

Cheers, Chris.

Aptivate | http://www.aptivate.org | Phone: +44 1223 760887
The Humanitarian Centre, Fenner's, Gresham Road, Cambridge CB1 2ES

Aptivate is a not-for-profit company registered in England and Wales
with company number 04980791.

#14

Hi Chris,

Can you add me as a Developer to your AppEngine instance (va. Google's
dashboard) (I'm assuming you're running on AppEngine).
Also, if possible, please add me ( mitchellsundt@gmail.com ) as a SiteAdmin
to that instance within Aggregate.

Mitch

··· On Fri, Nov 25, 2011 at 1:09 AM, Chris Wilson wrote:

Hi Mitch and all,

On Mon, 14 Nov 2011, Chris Wilson wrote:

In Aggregate 1.0 and earlier, there was a bug that didn't flush the most

recently submitted data to Fusion Tables or Google Spreadsheet. Aggregate
1.0.1 fixed that. If you haven't already, upgrade to 1.0.1 and see if the
publishing issue still exists. If it does, please let me know or open an
issue for it.

Thanks for the heads up, will give it a try and let you know.

I'm afraid it doesn't work. I have a 1.0.1 instance with 38 instances. I
export this to Fusion Tables (even just the existing data, not streaming
new data) and it stops after sending 8 of them.

This is a real shame because we want to build a desktop data review system
(perhaps in Microsoft Access) and ideally I'd like it to be linked live to
an online system, but since Fusion Tables integration is broken, we can't
rely on it, and the only thing I can do is integrate with the unstable
interfaces of Collect itself. Similarly we can't use the easy visualisation
tools provided by Google because we can't get the data into Fusion Tables.

Is there anything I can do to help debug this in the next couple of days?

Cheers, Chris.

Aptivate | http://www.aptivate.org | Phone: +44 1223 760887
The Humanitarian Centre, Fenner's, Gresham Road, Cambridge CB1 2ES

Aptivate is a not-for-profit company registered in England and Wales
with company number 04980791.

--
Post: opendatakit@googlegroups.com
Unsubscribe: opendatakit+unsubscribe@**googlegroups.comopendatakit%2Bunsubscribe@googlegroups.com
Options: http://groups.google.com/**group/opendatakit?hl=enhttp://groups.google.com/group/opendatakit?hl=en

--
Mitch Sundt
Software Engineer
University of Washington
mitchellsundt@gmail.com

#15

Hello,

Thanks for the speedy response.

I have had a quick look at that URL, but cannot find sufficient
information, in particular to understand where to submit the form to.
Perhaps it is there, and I have failed to see it. Similarly, when I
searched for things along the line of "bulk upload of ODK submissions using
curl", I was unable to find a single comprehensive explanation. I mention
this because I suspect that I am not the only person to hit this, and it
might be nice to have a single tutorial on how to feed submissions in
programmatically from a UNIX command line. I will probably create such a
resource via a blog post.

To summarise the issues I have encountered:

  1. I could not find information on how to programatically upload
    submissions, so had to look at the source code for the "upload submission
    page". This wasn't too hard.
  2. I wrote a simple curl script that worked for a while, but when I tried
    to put it on the production machine, it kept failing with authentication
    errors.
  3. In the URL you have kindly provided, I was not able to find sufficient
    information to understand how to upload submissions to a given instance,
    e.g., what the relative URL would be on a typical instance.
  4. Briefcase does not allow you to point it to a directory of XML
    submission files and have them upload to Aggregate.
  5. The documentation for briefcase indicates that even if this were
    possible, it would result in duplicates if the same submission is uploaded
    twice, whereas the "upload submission" web page correctly de-duplicates
    submissions, making it much easier to manage.

Using tcpdump I traced the authentication process, and discovered
sufficient information to allow me to adapt my script to first authenticate
to Aggregate, and then upload an arbitrary set of submissions via the
"upload submissions" page, with automatic de-duplication. I have thus
found a solution to my problem.

The script is available at
https://github.com/servalproject/succinct-data-example-config-bundles/blob/master/push-to-odk-aggregate/push-data-from-phone

All it requires is a file odkcredentials.txt containing username:password,
and a directory data/ containing the submissions to upload.

Paul.

··· On Thu, Sep 11, 2014 at 9:58 AM, Yaw Anokwa wrote:

Moving to developer list.

The thread you are responding to is a few years old. See
https://bitbucket.org/javarosa/javarosa/wiki/FormSubmissionAPI for the
API to implement.

What problem are you having with that API?

Yaw

Need ODK services? http://nafundi.com provides form design, server
setup, professional support, and software development for ODK.

On Wed, Sep 10, 2014 at 3:14 PM, paul.gardner.stephen@gmail.com wrote:

Hello,

What headers did you have to add? We are having the same problem.

Paul.

--

Post: opendatakit@googlegroups.com
Unsubscribe: opendatakit+unsubscribe@googlegroups.com
Options: http://groups.google.com/group/opendatakit?hl=en

You received this message because you are subscribed to the Google
Groups "ODK Community" group.
To unsubscribe from this group and stop receiving emails from it, send
an email to opendatakit+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

#16

chris,

i finally got a chance to try this out with a galaxy tab, and you are correct.

some samsung devices don't return the correct path to the sd card when
we call ask for an external storage location. instead, they return a
path to the internal storage location. see


for more.

the problem here isn't odk, it's samsung. it's a very easy fix, but
i'm not excited about putting device specific code into odk collect.
i've filed a bug report at
http://code.google.com/p/opendatakit/issues/detail?id=447.

yaw

··· On Tue, Nov 15, 2011 at 22:55, Chris Wilson wrote: > Hi Mitch, > > On Tue, 15 Nov 2011, Mitch Sundt wrote: > >> What version of Android are you using? > > This is 2.2 (FROYO.XWJJ7) on a Samsung Galaxy Tab GT-P1000, and I've tried > 1.1.7-RC2 and trunk (with the new menu structure) and neither of them save > anything on the (removable) SD card. > > I noticed when using ADB to install our custom version (with the constraint > checks skipped on save) that it has the option to install an application to > the SD card instead of the device's main memory, and I'm wondering whether > the location of the application might affect where the data is saved? > > Thank you all for your help by the way! It's much appreciated. I hope to > give something back to the community. > > Cheers, Chris. > -- > Aptivate | http://www.aptivate.org | Phone: +44 1223 760887 > The Humanitarian Centre, Fenner's, Gresham Road, Cambridge CB1 2ES > > Aptivate is a not-for-profit company registered in England and Wales > with company number 04980791. > > -- > Post: opendatakit@googlegroups.com > Unsubscribe: opendatakit+unsubscribe@googlegroups.com > Options: http://groups.google.com/group/opendatakit?hl=en >
#17

Hello,

Oh, it is probably helpful if I actually explain why I am doing this.

Together with New Zealand Red Cross we are creating "Succinct Data", which
is an open-source system for taking selected fields from ODK submissions,
and automatically compressing them so that this succinct data from a
submission can be transmitted efficiently by SMS or satellite short-message
service.

The compression uses the XML form specification file, including the
constraints placed on fields to achieve compelling compression performance.

We can typically compress a submission to about 8% - 10% of its original
XML size, allowing forms with a couple of dozen questions to be compressed
into a single SMS message.

These compressed messages are automatically reconstituted into XML
submission files at the receiving end, and then pushed into an ODK
Aggregate instance -- hence my exploration of possibilities for
programmatically feeding bare submission XML files into Aggregate.

We will be presenting this work at GHTC in San Jose next month, and are
happy to share our work with anyone who might find use of it.

Paul.

··· On Thu, Sep 11, 2014 at 11:02 AM, Paul Gardner-Stephen < paul@servalproject.org> wrote:

Hello,

Thanks for the speedy response.

I have had a quick look at that URL, but cannot find sufficient
information, in particular to understand where to submit the form to.
Perhaps it is there, and I have failed to see it. Similarly, when I
searched for things along the line of "bulk upload of ODK submissions using
curl", I was unable to find a single comprehensive explanation. I mention
this because I suspect that I am not the only person to hit this, and it
might be nice to have a single tutorial on how to feed submissions in
programmatically from a UNIX command line. I will probably create such a
resource via a blog post.

To summarise the issues I have encountered:

  1. I could not find information on how to programatically upload
    submissions, so had to look at the source code for the "upload submission
    page". This wasn't too hard.
  2. I wrote a simple curl script that worked for a while, but when I tried
    to put it on the production machine, it kept failing with authentication
    errors.
  3. In the URL you have kindly provided, I was not able to find sufficient
    information to understand how to upload submissions to a given instance,
    e.g., what the relative URL would be on a typical instance.
  4. Briefcase does not allow you to point it to a directory of XML
    submission files and have them upload to Aggregate.
  5. The documentation for briefcase indicates that even if this were
    possible, it would result in duplicates if the same submission is uploaded
    twice, whereas the "upload submission" web page correctly de-duplicates
    submissions, making it much easier to manage.

Using tcpdump I traced the authentication process, and discovered
sufficient information to allow me to adapt my script to first authenticate
to Aggregate, and then upload an arbitrary set of submissions via the
"upload submissions" page, with automatic de-duplication. I have thus
found a solution to my problem.

The script is available at
https://github.com/servalproject/succinct-data-example-config-bundles/blob/master/push-to-odk-aggregate/push-data-from-phone

All it requires is a file odkcredentials.txt containing username:password,
and a directory data/ containing the submissions to upload.

Paul.

On Thu, Sep 11, 2014 at 9:58 AM, Yaw Anokwa yanokwa@nafundi.com wrote:

Moving to developer list.

The thread you are responding to is a few years old. See
https://bitbucket.org/javarosa/javarosa/wiki/FormSubmissionAPI for the
API to implement.

What problem are you having with that API?

Yaw

Need ODK services? http://nafundi.com provides form design, server
setup, professional support, and software development for ODK.

On Wed, Sep 10, 2014 at 3:14 PM, paul.gardner.stephen@gmail.com wrote:

Hello,

What headers did you have to add? We are having the same problem.

Paul.

--

Post: opendatakit@googlegroups.com
Unsubscribe: opendatakit+unsubscribe@googlegroups.com
Options: http://groups.google.com/group/opendatakit?hl=en

You received this message because you are subscribed to the Google
Groups "ODK Community" group.
To unsubscribe from this group and stop receiving emails from it, send
an email to opendatakit+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

#18

Hi Yaw,

··· On Tue, 22 Nov 2011, Yaw Anokwa wrote:

i finally got a chance to try this out with a galaxy tab, and you are
correct.

some samsung devices don't return the correct path to the sd card when
we call ask for an external storage location. instead, they return a
path to the internal storage location. see
http://stackoverflow.com/questions/4210154/android-sd-card-characteristics-on-samsung-galaxy
for more.

the problem here isn't odk, it's samsung. it's a very easy fix, but i'm
not excited about putting device specific code into odk collect. i've
filed a bug report at
http://code.google.com/p/opendatakit/issues/detail?id=447.

Thanks for that :slight_smile: I figured it was something like this. /mnt/sdcard
appears to be the internal storage for some reason. It's not a showstopper
for us.

Cheers, Chris.

Aptivate | http://www.aptivate.org | Phone: +44 1223 760887
The Humanitarian Centre, Fenner's, Gresham Road, Cambridge CB1 2ES

Aptivate is a not-for-profit company registered in England and Wales
with company number 04980791.

#19

Not sure how this got attached to this old group thread....

The submission API is described here:

https://bitbucket.org/javarosa/javarosa/wiki/OpenRosaAPI

In particular, the submission protocol is described here:
https://bitbucket.org/javarosa/javarosa/wiki/FormSubmissionAPI

All interactions with the server assume that certain header fields are
being set appropriately, as described here:
https://bitbucket.org/javarosa/javarosa/wiki/OpenRosaRequest

Authentication uses DigestAuth in the standard
https://bitbucket.org/javarosa/javarosa/wiki/AuthenticationAPI though ODK
Aggregate also supports Google account authentication (this may break soon
with some changes Google has afoot).

The URL for ODK Aggregate is always:

In addition to these APIs, if you need to extract data from ODK Aggregate,
the data extraction API is detailed here:
http://code.google.com/p/opendatakit/wiki/BriefcaseAggregateAPI

Note that ODK Aggregate does not fully reconstruct an accurate XML
representation of the incoming submission -- namespace information, in
particular, is not preserved.

··· On Wed, Sep 10, 2014 at 11:27 PM, Paul Gardner-Stephen < paul@servalproject.org> wrote:

Hello,

Oh, it is probably helpful if I actually explain why I am doing this.

Together with New Zealand Red Cross we are creating "Succinct Data", which
is an open-source system for taking selected fields from ODK submissions,
and automatically compressing them so that this succinct data from a
submission can be transmitted efficiently by SMS or satellite short-message
service.

The compression uses the XML form specification file, including the
constraints placed on fields to achieve compelling compression performance.

We can typically compress a submission to about 8% - 10% of its original
XML size, allowing forms with a couple of dozen questions to be compressed
into a single SMS message.

These compressed messages are automatically reconstituted into XML
submission files at the receiving end, and then pushed into an ODK
Aggregate instance -- hence my exploration of possibilities for
programmatically feeding bare submission XML files into Aggregate.

We will be presenting this work at GHTC in San Jose next month, and are
happy to share our work with anyone who might find use of it.

Paul.

On Thu, Sep 11, 2014 at 11:02 AM, Paul Gardner-Stephen < paul@servalproject.org> wrote:

Hello,

Thanks for the speedy response.

I have had a quick look at that URL, but cannot find sufficient
information, in particular to understand where to submit the form to.
Perhaps it is there, and I have failed to see it. Similarly, when I
searched for things along the line of "bulk upload of ODK submissions using
curl", I was unable to find a single comprehensive explanation. I mention
this because I suspect that I am not the only person to hit this, and it
might be nice to have a single tutorial on how to feed submissions in
programmatically from a UNIX command line. I will probably create such a
resource via a blog post.

To summarise the issues I have encountered:

  1. I could not find information on how to programatically upload
    submissions, so had to look at the source code for the "upload submission
    page". This wasn't too hard.
  2. I wrote a simple curl script that worked for a while, but when I tried
    to put it on the production machine, it kept failing with authentication
    errors.
  3. In the URL you have kindly provided, I was not able to find sufficient
    information to understand how to upload submissions to a given instance,
    e.g., what the relative URL would be on a typical instance.
  4. Briefcase does not allow you to point it to a directory of XML
    submission files and have them upload to Aggregate.
  5. The documentation for briefcase indicates that even if this were
    possible, it would result in duplicates if the same submission is uploaded
    twice, whereas the "upload submission" web page correctly de-duplicates
    submissions, making it much easier to manage.

Using tcpdump I traced the authentication process, and discovered
sufficient information to allow me to adapt my script to first authenticate
to Aggregate, and then upload an arbitrary set of submissions via the
"upload submissions" page, with automatic de-duplication. I have thus
found a solution to my problem.

The script is available at
https://github.com/servalproject/succinct-data-example-config-bundles/blob/master/push-to-odk-aggregate/push-data-from-phone

All it requires is a file odkcredentials.txt containing
username:password, and a directory data/ containing the submissions to
upload.

Paul.

On Thu, Sep 11, 2014 at 9:58 AM, Yaw Anokwa yanokwa@nafundi.com wrote:

Moving to developer list.

The thread you are responding to is a few years old. See
https://bitbucket.org/javarosa/javarosa/wiki/FormSubmissionAPI for the
API to implement.

What problem are you having with that API?

Yaw

Need ODK services? http://nafundi.com provides form design, server
setup, professional support, and software development for ODK.

On Wed, Sep 10, 2014 at 3:14 PM, paul.gardner.stephen@gmail.com wrote:

Hello,

What headers did you have to add? We are having the same problem.

Paul.

--

Post: opendatakit@googlegroups.com
Unsubscribe: opendatakit+unsubscribe@googlegroups.com
Options: http://groups.google.com/group/opendatakit?hl=en

You received this message because you are subscribed to the Google
Groups "ODK Community" group.
To unsubscribe from this group and stop receiving emails from it, send
an email to opendatakit+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups
"ODK Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to opendatakit-developers+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--
Mitch Sundt
Software Engineer
University of Washington
mitchellsundt@gmail.com