Accept CSEntry as an optional Data client for ODK servers

1. What is the general goal of the feature?
Having relational data from questionnaire on device and on server can simplify greatly data engineers and data scientists duties. Usually, connect whatever software (GIS and statistical software) to relational database, Right now the ODK1 and 2 have multiples limitations, among others, regarding distributed synchronization and storage, programming, relational database support for questionnaire data on device and on server. CSEntry is the oldest CAPI software : come from more than 30 years of software development. And, right now, it have the most advanced functionalities and features know on any CAPI platform, among others, support of relational database for questionnaire data, support of sql on data collection application etc. However CSPro ecosystem doesn’t have any dashboard for data management and visualization (map, graph etc. ODK aggregate through several (ONA, KoBoToolBox etc) is very strong for dashboard, but weak in data collection.

2. What are some example use cases for this feature?
This will allow users to use several’s existing functions librairies writing in CSPro Language and facilitate task that are impossible to perform right now on ODK1 and 2. Supporting CSEntry will also facilitate client who support relational database data can allow a direct way to access to raw data during fieldwork. So users can perform live analysis and design script for automatic data processing using whatever languages: SQL, R, CPL (CSPro Programming Language) and Python…
Distributed storage and sync is also a must since CSEntry can sync to FTP, Dropbox. And there's no limitation for CSEntry on how manies servers that a device can sync to.
So, this will allow us to benefit best of two world. The power, flexibility, stability of CSPro and the best thinking dashboard, data management features from ODK1 and 2. So the the a more dynamic of users communities from ODK. *

3. What can you contribute to making this feature a reality?
Having CSEntry as a data client to ODK ecosystem will be not difficult since CSEntry already have an API: CSWeb. This API already have java librairies (https://github.com/IstatCooperation/CSPro2sql) able to connect to it and transfer questionnaire data in a specif relational schema on SQL server and MySQL right now. It will not difficult to adapt theses librairies to PostgreSQL/PostGIS, the RDBMS that is used ODK server. As a 16 year old user of CSEntry, MySQL and PostgreSQL, i can help for this project.

Hello and welcome, @htuser!

The integration as you have described it sounds like it exists entirely outside of any of the existing ODK tools. It also sounds like something that you are interested in building yourself. You are more than welcome to do this if you would like. As you have pointed out, all the relevant standards are public.

For examples of somewhat similar types of integrations that exist outside of core ODK tools, see @Enrico_Ferreguti's QGis plugin at QGisODK plugin or @Florian_May's ruODK at Seeking your feedback: ruODK. ODK is designed to focus on the collection of data so those integrations get ODK data out to analysis tools but if for some reason it would be helpful to pull data from other systems into ODK tools, that would be similarly possible.

Dear @LN,
I read somewhere on the forum a topic where @yanokwa list things, users request that we will never see on ODK... collect and servers...
I'm also reading several others posts from ODK users about limitation or things that are very easy to perform on CSEntry, but complicated or impossible on ODK 1 and 2. Among others:

  1. Distributed /simultaneous synchronization on multiple servers (Web, FTP, Cloud etc);
  2. Import of data for longitudinal survey;
  3. Full multi-language support for either variable, internal messages, labels on SPSS, STATA, R etc;
  4. MS Word/HTML/PDF like printing of filled questionnaire;
  5. Relational database support on devices and on server (relational schema for each application or questionnaire);
  6. Transparent encryption of data on an user defined basis: ability for each application developer to define his own encryption key/password for data;
  7. A more transparent way to define user defined functions in application;
  8. Best user managements through password management policy;
  9. Full support of lookup files;

So, i'm saying myself, rather than reinvent them or private ODK users to them, the ODK Developer Team should think about using CSEntry as an optional data client who can allow, more advanced users to overwrite manies of ODK 1 and 2 limitations. So, it's very difficult to do it without your agreements. However, normally this appear not so complicated since you have a RDBMS as backend and a CSEntry application can, through his API: CSWeb send, replicate or sync data on RDBMS such as MySQL/MariaDB, MS SQL server.

Also, i think, even if CSEntry is in Public Domain, its source code is not openly available on internet, notable exception for the CSWeb API. However, it's more easy for the Census Bureau to share it with a very well know institutes such as you.

Please continue to let me know what do you thing about!

1 Like