Programmatically retrieving data from ODK Aggregate

Hi all,

I'm new to ODK but after doing a fair bit of googling I can't find a
solution, so I figured I'd ask here. I have an application that needs to be
able to programmatically retrieve data from an instance of ODK Aggregate.
Ideally this would be done on an automated schedule, so having a way to get
the delta of submissions since last retrieval would be great. The data can
be in any format - JSON, XML, CSV, whatevs.

I've seen that there have been some mentions of publishing the data from
Aggregate to Google Fusion Tables, and then accessing the data through
Google's API, but that seems pretty Rube Goldberg-ish. I've also read that
ODK supports something called the OpenRosa API, but the terminology in that
documentation is confusing and I'm not sure if it provides the
functionality I'm looking for - advice would be appreciated here. Also,
I've seen ODK Briefcase, but as this is a standalone jar that still needs
to be triggered manually, it pretty much defeats my purpose.

Surely there has to be some way to do this - looking forward to your
suggestions.

Thanks,
Alan

There are several other options:

(1) if you have a server, you could set up a publisher (e.g., JSON
publisher) in ODK Aggregate to stream data to that server. See
http://code.google.com/p/opendatakit/wiki/AggregateToJSonXmlREDCapPublishers

(2) you can use ODK Briefcase to download the contents of ODK Aggregate,
emit a CSV and load that into your app.

(3) your app could implement the functionality of ODK Briefcase to access
the data directly from ODK Aggregate. See
http://code.google.com/p/opendatakit/wiki/BriefcaseAggregateAPI

··· On Tue, Nov 5, 2013 at 1:27 PM, Alan DeLong wrote:

Hi all,

I'm new to ODK but after doing a fair bit of googling I can't find a
solution, so I figured I'd ask here. I have an application that needs to be
able to programmatically retrieve data from an instance of ODK Aggregate.
Ideally this would be done on an automated schedule, so having a way to get
the delta of submissions since last retrieval would be great. The data can
be in any format - JSON, XML, CSV, whatevs.

I've seen that there have been some mentions of publishing the data from
Aggregate to Google Fusion Tables, and then accessing the data through
Google's API, but that seems pretty Rube Goldberg-ish. I've also read that
ODK supports something called the OpenRosa API, but the terminology in that
documentation is confusing and I'm not sure if it provides the
functionality I'm looking for - advice would be appreciated here. Also,
I've seen ODK Briefcase, but as this is a standalone jar that still needs
to be triggered manually, it pretty much defeats my purpose.

Surely there has to be some way to do this - looking forward to your
suggestions.

Thanks,
Alan

--
You received this message because you are subscribed to the Google Groups
"ODK Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to opendatakit-developers+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

--
Mitch Sundt
Software Engineer
University of Washington
mitchellsundt@gmail.com

Thanks Mitch, the JSON publisher seems to be exactly what I'm looking for.
The ODK instance the organization is running will require an upgrade and
then we should be all set.

Cheers,
Alan

··· On Tuesday, November 5, 2013 8:34:15 PM UTC-5, Mitch wrote: > > There are several other options: > > (1) if you have a server, you could set up a publisher (e.g., JSON > publisher) in ODK Aggregate to stream data to that server. See > http://code.google.com/p/opendatakit/wiki/AggregateToJSonXmlREDCapPublishers > > (2) you can use ODK Briefcase to download the contents of ODK Aggregate, > emit a CSV and load that into your app. > > (3) your app could implement the functionality of ODK Briefcase to access > the data directly from ODK Aggregate. See > http://code.google.com/p/opendatakit/wiki/BriefcaseAggregateAPI > > > > > On Tue, Nov 5, 2013 at 1:27 PM, Alan DeLong <aland...@gmail.com wrote: > >> Hi all, >> >> I'm new to ODK but after doing a fair bit of googling I can't find a >> solution, so I figured I'd ask here. I have an application that needs to be >> able to programmatically retrieve data from an instance of ODK Aggregate. >> Ideally this would be done on an automated schedule, so having a way to get >> the delta of submissions since last retrieval would be great. The data can >> be in any format - JSON, XML, CSV, whatevs. >> >> I've seen that there have been some mentions of publishing the data from >> Aggregate to Google Fusion Tables, and then accessing the data through >> Google's API, but that seems pretty Rube Goldberg-ish. I've also read that >> ODK supports something called the OpenRosa API, but the terminology in that >> documentation is confusing and I'm not sure if it provides the >> functionality I'm looking for - advice would be appreciated here. Also, >> I've seen ODK Briefcase, but as this is a standalone jar that still needs >> to be triggered manually, it pretty much defeats my purpose. >> >> Surely there has to be some way to do this - looking forward to your >> suggestions. >> >> Thanks, >> Alan >> >> -- >> You received this message because you are subscribed to the Google Groups >> "ODK Developers" group. >> To unsubscribe from this group and stop receiving emails from it, send an >> email to opendatakit-developers+unsubscribe@googlegroups.com >> . >> For more options, visit https://groups.google.com/groups/opt_out. >> > > > > -- > Mitch Sundt > Software Engineer > University of Washington > mitche...@gmail.com >

Mitch,
I am also interested in experimenting with a JSON publisher. It appears
that the github link (github.com/eschluntz/odk_receiver) in the Simple JSON
publisher section of the wiki page is not longer working. Is this or a
similar example available elsewhere on GitHub?
Thanks,
Steve

··· On Tuesday, November 5, 2013 5:34:15 PM UTC-8, Mitch wrote: > > There are several other options: > > (1) if you have a server, you could set up a publisher (e.g., JSON > publisher) in ODK Aggregate to stream data to that server. See > http://code.google.com/p/opendatakit/wiki/AggregateToJSonXmlREDCapPublishers > > (2) you can use ODK Briefcase to download the contents of ODK Aggregate, > emit a CSV and load that into your app. > > (3) your app could implement the functionality of ODK Briefcase to access > the data directly from ODK Aggregate. See > http://code.google.com/p/opendatakit/wiki/BriefcaseAggregateAPI > > > > > On Tue, Nov 5, 2013 at 1:27 PM, Alan DeLong <aland...@gmail.com wrote: > >> Hi all, >> >> I'm new to ODK but after doing a fair bit of googling I can't find a >> solution, so I figured I'd ask here. I have an application that needs to be >> able to programmatically retrieve data from an instance of ODK Aggregate. >> Ideally this would be done on an automated schedule, so having a way to get >> the delta of submissions since last retrieval would be great. The data can >> be in any format - JSON, XML, CSV, whatevs. >> >> I've seen that there have been some mentions of publishing the data from >> Aggregate to Google Fusion Tables, and then accessing the data through >> Google's API, but that seems pretty Rube Goldberg-ish. I've also read that >> ODK supports something called the OpenRosa API, but the terminology in that >> documentation is confusing and I'm not sure if it provides the >> functionality I'm looking for - advice would be appreciated here. Also, >> I've seen ODK Briefcase, but as this is a standalone jar that still needs >> to be triggered manually, it pretty much defeats my purpose. >> >> Surely there has to be some way to do this - looking forward to your >> suggestions. >> >> Thanks, >> Alan >> >> -- >> You received this message because you are subscribed to the Google Groups >> "ODK Developers" group. >> To unsubscribe from this group and stop receiving emails from it, send an >> email to opendatakit-developers+unsubscribe@googlegroups.com >> . >> For more options, visit https://groups.google.com/groups/opt_out. >> > > > > -- > Mitch Sundt > Software Engineer > University of Washington > mitche...@gmail.com >

Perhaps someone in the community can answer this?

I don't know of one. When I need to, I just set up a port listener and spew
the transmission into a console window for testing purposes.

··· On Tue, Jan 28, 2014 at 10:41 AM, Steven Hansen wrote:

Mitch,
I am also interested in experimenting with a JSON publisher. It appears
that the github link (github.com/eschluntz/odk_receiver) in the Simple
JSON publisher section of the wiki page is not longer working. Is this or a
similar example available elsewhere on GitHub?
Thanks,
Steve

On Tuesday, November 5, 2013 5:34:15 PM UTC-8, Mitch wrote:

There are several other options:

(1) if you have a server, you could set up a publisher (e.g., JSON
publisher) in ODK Aggregate to stream data to that server. See
http://code.google.com/p/opendatakit/wiki/AggregateToJSonXmlREDCapPublis
hers

(2) you can use ODK Briefcase to download the contents of ODK Aggregate,
emit a CSV and load that into your app.

(3) your app could implement the functionality of ODK Briefcase to access
the data directly from ODK Aggregate. See http://code.google.com/p/
opendatakit/wiki/BriefcaseAggregateAPI

On Tue, Nov 5, 2013 at 1:27 PM, Alan DeLong aland...@gmail.com wrote:

Hi all,

I'm new to ODK but after doing a fair bit of googling I can't find a
solution, so I figured I'd ask here. I have an application that needs to be
able to programmatically retrieve data from an instance of ODK Aggregate.
Ideally this would be done on an automated schedule, so having a way to get
the delta of submissions since last retrieval would be great. The data can
be in any format - JSON, XML, CSV, whatevs.

I've seen that there have been some mentions of publishing the data from
Aggregate to Google Fusion Tables, and then accessing the data through
Google's API, but that seems pretty Rube Goldberg-ish. I've also read that
ODK supports something called the OpenRosa API, but the terminology in that
documentation is confusing and I'm not sure if it provides the
functionality I'm looking for - advice would be appreciated here. Also,
I've seen ODK Briefcase, but as this is a standalone jar that still needs
to be triggered manually, it pretty much defeats my purpose.

Surely there has to be some way to do this - looking forward to your
suggestions.

Thanks,
Alan

--
You received this message because you are subscribed to the Google
Groups "ODK Developers" group.
To unsubscribe from this group and stop receiving emails from it, send
an email to opendatakit-developers+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

--
Mitch Sundt
Software Engineer
University of Washington
mitche...@gmail.com

--
You received this message because you are subscribed to the Google Groups
"ODK Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to opendatakit-developers+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

--
Mitch Sundt
Software Engineer
University of Washington
mitchellsundt@gmail.com

I guess I have a related, more basic question.
In ODK Aggregate, I am selecting Publish, and choosing Z-ALPHA JSON Server
and Stream new Submission Data ONLY

I suppose I need more detailed instructions on setting up the URL to
publish to and the Authorization token. I don't have experience with JSON
data, so perhaps this is more of a question about the basics of JSON data
rather than ODK.

Thanks again.

··· On Tue, Jan 28, 2014 at 10:58 AM, Mitch Sundt wrote:

Perhaps someone in the community can answer this?

I don't know of one. When I need to, I just set up a port listener and
spew the transmission into a console window for testing purposes.

On Tue, Jan 28, 2014 at 10:41 AM, Steven Hansen steveohansen@gmail.comwrote:

Mitch,
I am also interested in experimenting with a JSON publisher. It appears
that the github link (github.com/eschluntz/odk_receiver) in the Simple
JSON publisher section of the wiki page is not longer working. Is this or a
similar example available elsewhere on GitHub?
Thanks,
Steve

On Tuesday, November 5, 2013 5:34:15 PM UTC-8, Mitch wrote:

There are several other options:

(1) if you have a server, you could set up a publisher (e.g., JSON
publisher) in ODK Aggregate to stream data to that server. See
http://code.google.com/p/opendatakit/wiki/AggregateToJSonXmlREDCapPublis
hers

(2) you can use ODK Briefcase to download the contents of ODK Aggregate,
emit a CSV and load that into your app.

(3) your app could implement the functionality of ODK Briefcase to
access the data directly from ODK Aggregate. See
http://code.google.com/p/opendatakit/wiki/BriefcaseAggregateAPI

On Tue, Nov 5, 2013 at 1:27 PM, Alan DeLong aland...@gmail.com wrote:

Hi all,

I'm new to ODK but after doing a fair bit of googling I can't find a
solution, so I figured I'd ask here. I have an application that needs to be
able to programmatically retrieve data from an instance of ODK Aggregate.
Ideally this would be done on an automated schedule, so having a way to get
the delta of submissions since last retrieval would be great. The data can
be in any format - JSON, XML, CSV, whatevs.

I've seen that there have been some mentions of publishing the data
from Aggregate to Google Fusion Tables, and then accessing the data through
Google's API, but that seems pretty Rube Goldberg-ish. I've also read that
ODK supports something called the OpenRosa API, but the terminology in that
documentation is confusing and I'm not sure if it provides the
functionality I'm looking for - advice would be appreciated here. Also,
I've seen ODK Briefcase, but as this is a standalone jar that still needs
to be triggered manually, it pretty much defeats my purpose.

Surely there has to be some way to do this - looking forward to your
suggestions.

Thanks,
Alan

--
You received this message because you are subscribed to the Google
Groups "ODK Developers" group.
To unsubscribe from this group and stop receiving emails from it, send
an email to opendatakit-developers+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

--
Mitch Sundt
Software Engineer
University of Washington
mitche...@gmail.com

--
You received this message because you are subscribed to the Google Groups
"ODK Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to opendatakit-developers+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

--
Mitch Sundt
Software Engineer
University of Washington
mitchellsundt@gmail.com

--
You received this message because you are subscribed to a topic in the
Google Groups "ODK Developers" group.
To unsubscribe from this topic, visit
https://groups.google.com/d/topic/opendatakit-developers/54eHFTwshG8/unsubscribe
.
To unsubscribe from this group and all its topics, send an email to
opendatakit-developers+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

Steven,

You need to have a server that listens for JSON formatted submissions
at the URL you enter. And to ensure that you only get posts that you
want, you can add some authorization token (basically a password that
you check for your server).

The precise format of the JSON is at
https://code.google.com/p/opendatakit/wiki/AggregateToJSonXmlREDCapPublishers.

If you want a quick and dirty server to test with, attached is the
source for a JSON server I hacked together. It listens for posts on
port 12345 and prints the result out to the command line.

Yaw

server.py (557 Bytes)

··· On Tue, Jan 28, 2014 at 12:08 PM, Steven Hansen wrote: > I guess I have a related, more basic question. > In ODK Aggregate, I am selecting Publish, and choosing Z-ALPHA JSON Server > and Stream new Submission Data ONLY > > I suppose I need more detailed instructions on setting up the URL to publish > to and the Authorization token. I don't have experience with JSON data, so > perhaps this is more of a question about the basics of JSON data rather than > ODK. > > Thanks again. > > > On Tue, Jan 28, 2014 at 10:58 AM, Mitch Sundt wrote: >> >> Perhaps someone in the community can answer this? >> >> I don't know of one. When I need to, I just set up a port listener and >> spew the transmission into a console window for testing purposes. >> >> >> On Tue, Jan 28, 2014 at 10:41 AM, Steven Hansen wrote: >>> >>> Mitch, >>> I am also interested in experimenting with a JSON publisher. It appears >>> that the github link (github.com/eschluntz/odk_receiver) in the Simple JSON >>> publisher section of the wiki page is not longer working. Is this or a >>> similar example available elsewhere on GitHub? >>> Thanks, >>> Steve >>> >>> On Tuesday, November 5, 2013 5:34:15 PM UTC-8, Mitch wrote: >>>> >>>> There are several other options: >>>> >>>> (1) if you have a server, you could set up a publisher (e.g., JSON >>>> publisher) in ODK Aggregate to stream data to that server. See >>>> http://code.google.com/p/opendatakit/wiki/AggregateToJSonXmlREDCapPublishers >>>> >>>> (2) you can use ODK Briefcase to download the contents of ODK Aggregate, >>>> emit a CSV and load that into your app. >>>> >>>> (3) your app could implement the functionality of ODK Briefcase to >>>> access the data directly from ODK Aggregate. See >>>> http://code.google.com/p/opendatakit/wiki/BriefcaseAggregateAPI >>>> >>>> >>>> >>>> >>>> On Tue, Nov 5, 2013 at 1:27 PM, Alan DeLong wrote: >>>>> >>>>> Hi all, >>>>> >>>>> I'm new to ODK but after doing a fair bit of googling I can't find a >>>>> solution, so I figured I'd ask here. I have an application that needs to be >>>>> able to programmatically retrieve data from an instance of ODK Aggregate. >>>>> Ideally this would be done on an automated schedule, so having a way to get >>>>> the delta of submissions since last retrieval would be great. The data can >>>>> be in any format - JSON, XML, CSV, whatevs. >>>>> >>>>> I've seen that there have been some mentions of publishing the data >>>>> from Aggregate to Google Fusion Tables, and then accessing the data through >>>>> Google's API, but that seems pretty Rube Goldberg-ish. I've also read that >>>>> ODK supports something called the OpenRosa API, but the terminology in that >>>>> documentation is confusing and I'm not sure if it provides the functionality >>>>> I'm looking for - advice would be appreciated here. Also, I've seen ODK >>>>> Briefcase, but as this is a standalone jar that still needs to be triggered >>>>> manually, it pretty much defeats my purpose. >>>>> >>>>> Surely there has to be some way to do this - looking forward to your >>>>> suggestions. >>>>> >>>>> Thanks, >>>>> Alan >>>>> >>>>> -- >>>>> You received this message because you are subscribed to the Google >>>>> Groups "ODK Developers" group. >>>>> To unsubscribe from this group and stop receiving emails from it, send >>>>> an email to opendatakit-developers+unsubscribe@googlegroups.com. >>>>> For more options, visit https://groups.google.com/groups/opt_out. >>>> >>>> >>>> >>>> >>>> -- >>>> Mitch Sundt >>>> Software Engineer >>>> University of Washington >>>> mitche...@gmail.com >>> >>> -- >>> You received this message because you are subscribed to the Google Groups >>> "ODK Developers" group. >>> To unsubscribe from this group and stop receiving emails from it, send an >>> email to opendatakit-developers+unsubscribe@googlegroups.com. >>> For more options, visit https://groups.google.com/groups/opt_out. >> >> >> >> >> -- >> Mitch Sundt >> Software Engineer >> University of Washington >> mitchellsundt@gmail.com >> >> -- >> You received this message because you are subscribed to a topic in the >> Google Groups "ODK Developers" group. >> To unsubscribe from this topic, visit >> https://groups.google.com/d/topic/opendatakit-developers/54eHFTwshG8/unsubscribe. >> To unsubscribe from this group and all its topics, send an email to >> opendatakit-developers+unsubscribe@googlegroups.com. >> >> For more options, visit https://groups.google.com/groups/opt_out. > > > -- > You received this message because you are subscribed to the Google Groups > "ODK Developers" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to opendatakit-developers+unsubscribe@googlegroups.com. > For more options, visit https://groups.google.com/groups/opt_out.

And note that there are some bugs with the serializer around an extra comma
and strings containing quotes or commas not being properly escaped. These
will be fixed in the ODK Aggregate 1.4.1 release (soon).

··· On Tue, Jan 28, 2014 at 3:09 PM, Yaw Anokwa wrote:

Steven,

You need to have a server that listens for JSON formatted submissions
at the URL you enter. And to ensure that you only get posts that you
want, you can add some authorization token (basically a password that
you check for your server).

The precise format of the JSON is at

https://code.google.com/p/opendatakit/wiki/AggregateToJSonXmlREDCapPublishers
.

If you want a quick and dirty server to test with, attached is the
source for a JSON server I hacked together. It listens for posts on
port 12345 and prints the result out to the command line.

Yaw

On Tue, Jan 28, 2014 at 12:08 PM, Steven Hansen steveohansen@gmail.com wrote:

I guess I have a related, more basic question.
In ODK Aggregate, I am selecting Publish, and choosing Z-ALPHA JSON
Server
and Stream new Submission Data ONLY

I suppose I need more detailed instructions on setting up the URL to
publish
to and the Authorization token. I don't have experience with JSON data,
so
perhaps this is more of a question about the basics of JSON data rather
than
ODK.

Thanks again.

On Tue, Jan 28, 2014 at 10:58 AM, Mitch Sundt mitchellsundt@gmail.com wrote:

Perhaps someone in the community can answer this?

I don't know of one. When I need to, I just set up a port listener and
spew the transmission into a console window for testing purposes.

On Tue, Jan 28, 2014 at 10:41 AM, Steven Hansen <steveohansen@gmail.com wrote:

Mitch,
I am also interested in experimenting with a JSON publisher. It appears
that the github link (github.com/eschluntz/odk_receiver) in the
Simple JSON
publisher section of the wiki page is not longer working. Is this or a
similar example available elsewhere on GitHub?
Thanks,
Steve

On Tuesday, November 5, 2013 5:34:15 PM UTC-8, Mitch wrote:

There are several other options:

(1) if you have a server, you could set up a publisher (e.g., JSON
publisher) in ODK Aggregate to stream data to that server. See

http://code.google.com/p/opendatakit/wiki/AggregateToJSonXmlREDCapPublishers

(2) you can use ODK Briefcase to download the contents of ODK
Aggregate,
emit a CSV and load that into your app.

(3) your app could implement the functionality of ODK Briefcase to
access the data directly from ODK Aggregate. See
http://code.google.com/p/opendatakit/wiki/BriefcaseAggregateAPI

On Tue, Nov 5, 2013 at 1:27 PM, Alan DeLong aland...@gmail.com wrote:

Hi all,

I'm new to ODK but after doing a fair bit of googling I can't find a
solution, so I figured I'd ask here. I have an application that
needs to be
able to programmatically retrieve data from an instance of ODK
Aggregate.
Ideally this would be done on an automated schedule, so having a way
to get
the delta of submissions since last retrieval would be great. The
data can
be in any format - JSON, XML, CSV, whatevs.

I've seen that there have been some mentions of publishing the data
from Aggregate to Google Fusion Tables, and then accessing the data
through
Google's API, but that seems pretty Rube Goldberg-ish. I've also
read that
ODK supports something called the OpenRosa API, but the terminology
in that
documentation is confusing and I'm not sure if it provides the
functionality
I'm looking for - advice would be appreciated here. Also, I've seen
ODK
Briefcase, but as this is a standalone jar that still needs to be
triggered
manually, it pretty much defeats my purpose.

Surely there has to be some way to do this - looking forward to your
suggestions.

Thanks,
Alan

--
You received this message because you are subscribed to the Google
Groups "ODK Developers" group.
To unsubscribe from this group and stop receiving emails from it,
send
an email to opendatakit-developers+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

--
Mitch Sundt
Software Engineer
University of Washington
mitche...@gmail.com

--
You received this message because you are subscribed to the Google
Groups
"ODK Developers" group.
To unsubscribe from this group and stop receiving emails from it, send
an
email to opendatakit-developers+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

--
Mitch Sundt
Software Engineer
University of Washington
mitchellsundt@gmail.com

--
You received this message because you are subscribed to a topic in the
Google Groups "ODK Developers" group.
To unsubscribe from this topic, visit

https://groups.google.com/d/topic/opendatakit-developers/54eHFTwshG8/unsubscribe
.

To unsubscribe from this group and all its topics, send an email to
opendatakit-developers+unsubscribe@googlegroups.com.

For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google Groups
"ODK Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to opendatakit-developers+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google Groups
"ODK Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to opendatakit-developers+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

--
Mitch Sundt
Software Engineer
University of Washington
mitchellsundt@gmail.com