I have a server that can read in csv files and stores data. I was wondering if it was possible to have ODK collect directly send the forms in csv file format to that server. Is this the same as setting up ODK aggregate through the server?
Thanks!
Anjali
oops I meant the completed CSV attachments! how do you get it to send somewhere without uploading to aggregate? or how do I connect a server to collect to receive such csv files
I think a couple of options that might help are outlined in the documentation section on ODK Aggregate APIs. Let us know if that doesn't help!
I'm glad to see you've gotten a couple of other questions answered successfully here on the forums. We also encourage users to post an introduction to Introduce yourself here! and add a profile picture. Welcome to ODK!
2 Likes
AFAIK ODK Collect cannot 'submit' an inspection as a csv to a custom server API, but instead you can submit it (as XML) to ODK Aggregate, and then have Aggregate export them as csv's, which you can then import into your server.
Alternatively, rather than having to manually/periodically export csv's from Aggregate, you could setup Aggregate to stream (aka "Publish") incoming submissions over to your server. You'd need to convert them (from, say, JSON) to csv prior to import, but that can be easily scripted.
1 Like
Thank you! I will check them out and let you know if it helps
This is exactly what I need! I will try and set up aggregate to automatically stream the incoming submissions over to the server--I didn't realize you could automate conversion JSON to csv. Do you know any resources to do so?
Thank you!
Anjali
There are a number of online (ie web) tools that will do so, but probably you'll want some actual code on your server to do the conversion. A quick google search gave this, but I'm sure there are probably numerous others, depending on your preferred programming language.
You'll still need to setup a recipient on your server to receive the JSON-encoded submission stream coming from Aggregate, but a basic Apache PHP script can accomplish that, then convert the incoming JSON to cvs (see above) and then kick-off a DB import request. [I did similar to stream ODK Aggregate submissions into KML, to dynamically update a Google Earth view as inspections came in].
Thank you! I understand the conceptual steps that need to get done but quite confused how to proceed technically. I'm quite new to all the server stuff--are there any tutorials? I've been trying to google and all I've found are different posts from the forum with people having trouble with different issues
Thank you again!
I haven't ever tried something like this. I think you should be able to grab the data from Aggregate via the simple JSON publisher. And then on the server receiving the response you would need, as noted by @Xiphware, some custom code to convert the json to csv (for example, a node.js app that uses json2csv) and hand over the converted data to your server to read in the csv and store the data.
Kobo also has some API functionality that might be of interest to you.
ODK Aggregate has all sorts of ways of getting data out. https://docs.opendatakit.org/aggregate-data-access has a pretty good description.
If you'd like to try a more fast-moving server, ODK Central also has a great RESTful API!
First off, you'll need to setup something on your server to receive HTTP POST messages containing the JSON coming from ODK Aggregate. How to accomplish that will largely depend on what exactly your server is; eg a Linux box, Windows box, etc... Can you tell me what your server is?
I'm realized the server is an Apache FTP server, can you publish to an FTP server? Sorry I'm new to this and previously didn't know the distinctions between types of servers
Hi @akatta13, it is possible to do what you describe, but it will require a piece of software between ODK and the FTP server. All this to say, you'll need someone technical (likely a software developer) to make it work.
1 Like
From Apache ftpserver:
The Apache FtpServer is a 100% pure Java FTP server. It's designed to be a complete and portable FTP server engine solution based on currently available open protocols. FtpServer can be run standalone as a Windows service or Unix/Linux daemon, or embedded into a Java application.
I'd need to know what flavor your ftpserver is - Linux, Windows or embedded Java - before knowing how to recommend proceeding.
In the mean time, you can test that your published data stream from ODK Aggregate is working OK by sending it here, which will display your results as they come in. This service is described in more detail here: https://github.com/spatialthoughts/odk-post-receiver. This is a good first step to make sure your data is actually getting sent correctly; then you can focus on setting up a receiver on your server, and test its then getting received (with the knowledge that it is in fact being sent ok). Helps isolate where a problem lies when it don't work...
3 Likes
Thank you for all your help! The server is run through a standalone Windows service.
I've also been going through connecting it to the post receiver and it's been helpful to make sure that its working well.
thank you again 
I was afraid you'd say that... 
So basically yer probably going to have to setup a Windows-based webserver, of which you have a wide variety of options - eg IIS, Apache HTTP, Tomcat, lighttpd, ... - none of which are my forte, alas. So which is best is probably dependent on what local skillset you might have available to you (either your own, or of those around you). Or, alternatively, what your preferred programming/scripting language is - PHP, Python, Javascript, ... - which may determine what the most appropriate webserver is.
Once your webserver is setup and running, you'll need to write a script that'll get fired off when your webserver receives a POST message from your ODK Aggregate containing a JSON text string containing a submission data. First off I'd probably just dump this string into a temporary file, simply to make sure all the data is actually getting across OK. It should be a simple enough script in whatever language you choose, but it will take a little programming.
Once you confirm the data is getting there intact, you can run the JSON thru a json2csv converter and write it out to a csv file, in an appropriate location for your ftpserver to pick up.
2 Likes
Thank you so much! I'm going to have to ask some people around me to help out with this but I think I understand the steps to get there. Thanks again !
YW. Please keep us all posted on how you make out
1 Like
Please also note that the handshaking between ODK Aggregate and your webserver may not always be perfect, and sometimes a submission may get re-sent. Which is to say, your script should handle the situation where it receives an identical duplicate copy of something it previously received. There's more info about this here:
...Under certain failure conditions, the downstream service can receive multiple copies of a given submission. This is known, expected, behavior.