How to run ODK-Collect with Self-signed SSL certificate on the server

Hi ODK developers,

Thanks for your great work on ODK. We have forked ODK-Collect and we
connect it to our OpenXdata server.

One of the first problems that we faced while running ODK with OXD was that
we were using a self-signed SSL certificate on OXD which obviously didn't
work on ODK. So we tried and succeeded in getting it running. I realized
that this is a common problem that people face and hence wrote a blog post
to help other people out. I am sending this email to this group so that
people can find and refer to it at later times.

Here are the links:


This blog post is a preliminary post about SSL, MITM, self-signed
certificates and what are the various options when you want to run an
android app with a self-signed certificate on your server.


This is the link which contains all code changes you need to do to run
ODK-Collect with any server (not just OpenXdata) having a self-signed
certificate.

Hope this helps, and in case anybody has any issues, you can write comments
to the blog post directly or reply to this email, and I will try to help
you out.

Thanks and regards,
Raghu Mittal
Co-founder & Tech Lead
Handheld Solutions & Research Labs Pvt Ltd
handsrel.com

Hi Raghu,

Thanks for the tips you mentioned in the blogposts. I would like to
recompile an Androd app from *opendatakit *source but cannot find the
/res/raw/ inside the *androidapp *dir. I just forked the latest code as
at 30th Oct, 2014. Anything I am missing? Looking forward to your response.

Regards,
Caesar

Hello Raghu,

After a few hurdles and following you guidelines, I was able to get ODK
Collect (1.4.4 rev 1047) source code compiled and running on my android
phone. I have also managed to get it partly working with the self-signed
certificate and overcome certificate incompatibility error I was getting
before which was caused as you suggested by the needed modification. I have
installed OXD (OpenXData 1.16.7 (build 54)) and enabled access to port
443. However when I connect to the application from the ODK app, When
trying to load empty forms, I get a Form listing failed. Error:
org.opendatakit.httpclientandroidlib.conn.ConnectTimeoutException. Did you
experience any such issues at all? Does this imply some needed changes on
the ODK Collect code? Thanks in advance for your suggestions.

Regards,
Caesar Olima

Caesar, what tools are you using to compile? ant? eclipse? Where is it
telling you you need /res/raw? I've never seen that before. Can you post
the compile command you're giving and what errors you're getting?

··· On 30 October 2014 08:49, Caesar Kennedy Olima wrote:

Hi Raghu,

Thanks for the tips you mentioned in the blogposts. I would like to
recompile an Androd app from *opendatakit *source but cannot find the
/res/raw/ inside the *androidapp *dir. I just forked the latest code as
at 30th Oct, 2014. Anything I am missing? Looking forward to your response.

Regards,
Caesar

--
You received this message because you are subscribed to the Google Groups
"ODK Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to opendatakit-developers+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--
Tom Smyth

Worker-Owner, Sassafras Tech Collective
Specializing in innovative, usable tech for social change
sassafras.coop · @sassafrastech

Resident, Touchstone Cohousing
touchstonecohousing.org

Sounds like the URL you are connecting to is not responding, or, if you
have a debugger set up on the server, and have set a breakpoint there and
are stepping through it, the device may time out the connection because the
server is not responding and sending data back to the client. ODK Collect
has a 30-second connection timeout -- if the server does not start sending
bytes back to the client, ODK Collect will terminate the connection. The
setting is WebUtils.CONNECTION_TIMEOUT If you are debugging the server, you
will want to add a zero to that (5 min) so that you can step through the
server without ODK Collect terminating a connection. But you will want to
set this back down to 30 or 60 seconds so that users don't experience the
application "hanging" and "doing nothing" during network access.

··· On Tue, Nov 18, 2014 at 3:58 AM, Caesar K. Olima wrote:

Hello Raghu,

After a few hurdles and following you guidelines, I was able to get ODK
Collect (1.4.4 rev 1047) source code compiled and running on my
android phone. I have also managed to get it partly working with the
self-signed certificate and overcome certificate incompatibility error I
was getting before which was caused as you suggested by the needed
modification. I have installed OXD (OpenXData 1.16.7 (build 54)) and
enabled access to port 443. However when I connect to the application from
the ODK app, When trying to load empty forms, I get a Form listing failed.
Error:
org.opendatakit.httpclientandroidlib.conn.ConnectTimeoutException. Did you
experience any such issues at all? Does this imply some needed changes on
the ODK Collect code? Thanks in advance for your suggestions.

Regards,
Caesar Olima

--
You received this message because you are subscribed to the Google Groups
"ODK Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to opendatakit-developers+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--
Mitch Sundt
Software Engineer
University of Washington
mitchellsundt@gmail.com

Note also that some organizations can get free commercial SSL certificates
(so you don't need tocode in a self-signed one):

FREE SSL certificates are available for charities and non-profits
https://www.comodo.com/news/press_releases/10_01_06.html

FREE SSL certificates are available for open-source projects (probably not
applicable to most people):
https://www.globalsign.eu/company/press/061913-globalsign-offers-free-ssl-certificates-for-open-source-projects.html

··· On Thu, Oct 30, 2014 at 5:53 AM, Tom Smyth wrote:

Caesar, what tools are you using to compile? ant? eclipse? Where is it
telling you you need /res/raw? I've never seen that before. Can you post
the compile command you're giving and what errors you're getting?

On 30 October 2014 08:49, Caesar Kennedy Olima caesar.olima@gmail.com wrote:

Hi Raghu,

Thanks for the tips you mentioned in the blogposts. I would like to
recompile an Androd app from *opendatakit *source but cannot find the
/res/raw/ inside the *androidapp *dir. I just forked the latest code
as at 30th Oct, 2014. Anything I am missing? Looking forward to your
response.

Regards,
Caesar

--
You received this message because you are subscribed to the Google Groups
"ODK Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to opendatakit-developers+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--
Tom Smyth

Worker-Owner, Sassafras Tech Collective
Specializing in innovative, usable tech for social change
sassafras.coop · @sassafrastech

Resident, Touchstone Cohousing
touchstonecohousing.org

--
You received this message because you are subscribed to the Google Groups
"ODK Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to opendatakit-developers+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--
Mitch Sundt
Software Engineer
University of Washington
mitchellsundt@gmail.com

Hi Tom,

I am using Eclipse to compile the source code. According to instructions by
Raghu from the 2nd link (
http://thetechnohaven.blogspot.in/2014/05/self-signed-certificate-in-android.html),
in Step 2, one is supposed to copy the .keystore to
addroidappdir/res/raw/ was just wondering if we need to create this
directory as I cannot see the directory from the downloaded code. Is there
some steps I might have missed or some version/branch I was supposed to
clone?

Regards,
Caesar

Hi Tom,

I am using Eclipse to compile the source code. From Raghu's reply, seems
I'll have to create the mentioned directory. Thanks for asking.

regards,
caesar

Hi Caesar,

I suggest that you put this question on openxdata-developer group (
openxdata-dev@googlegroups.com) since there are many others running OXD
with ODK Collect and they will be able to help you out.

Since you are running the older OpenXdata build, it may be that the ODK
proto is missing. Can you check if the proto is available or not? Also, you
may have to pick up a much later build of OXD to get it to work. I am using
the OXD trunk (latest code) and it works pretty well with ODK Collect.

Regards,
Raghu
Co-founder & Tech Lead
Handheld Solutions & Research Labs Pvt Ltd
handsrel.com

··· On Tue, Nov 18, 2014 at 11:47 PM, Mitch Sundt wrote:

Sounds like the URL you are connecting to is not responding, or, if you
have a debugger set up on the server, and have set a breakpoint there and
are stepping through it, the device may time out the connection because the
server is not responding and sending data back to the client. ODK Collect
has a 30-second connection timeout -- if the server does not start sending
bytes back to the client, ODK Collect will terminate the connection. The
setting is WebUtils.CONNECTION_TIMEOUT If you are debugging the server, you
will want to add a zero to that (5 min) so that you can step through the
server without ODK Collect terminating a connection. But you will want to
set this back down to 30 or 60 seconds so that users don't experience the
application "hanging" and "doing nothing" during network access.

On Tue, Nov 18, 2014 at 3:58 AM, Caesar K. Olima caesar.olima@gmail.com wrote:

Hello Raghu,

After a few hurdles and following you guidelines, I was able to get ODK
Collect (1.4.4 rev 1047) source code compiled and running on my
android phone. I have also managed to get it partly working with the
self-signed certificate and overcome certificate incompatibility error I
was getting before which was caused as you suggested by the needed
modification. I have installed OXD (OpenXData 1.16.7 (build 54)) and
enabled access to port 443. However when I connect to the application from
the ODK app, When trying to load empty forms, I get a Form listing failed.
Error:
org.opendatakit.httpclientandroidlib.conn.ConnectTimeoutException. Did you
experience any such issues at all? Does this imply some needed changes on
the ODK Collect code? Thanks in advance for your suggestions.

Regards,
Caesar Olima

--
You received this message because you are subscribed to the Google Groups
"ODK Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to opendatakit-developers+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--
Mitch Sundt
Software Engineer
University of Washington
mitchellsundt@gmail.com

--
You received this message because you are subscribed to a topic in the
Google Groups "ODK Developers" group.
To unsubscribe from this topic, visit
https://groups.google.com/d/topic/opendatakit-developers/UMPyRcgQq1k/unsubscribe
.
To unsubscribe from this group and all its topics, send an email to
opendatakit-developers+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Hello Mitch,

Thanks for the suggestions on acquiring free SSL certificates. I wasn't
aware of these. I am performing these tests on behalf of ,KEMRI Wellcome
Trust, a non-profit making organisation in Kenya. It is worth knowing this
option exists so maybe we can take advantage of it at a later time. Thanks
once again.

Kind Regards,
Caesar

Ok good luck.

··· On 31 October 2014 03:26, Caesar K. Olima wrote:

Hi Tom,

I am using Eclipse to compile the source code. From Raghu's reply, seems
I'll have to create the mentioned directory. Thanks for asking.

regards,
caesar

--
You received this message because you are subscribed to the Google Groups
"ODK Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to opendatakit-developers+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--
Tom Smyth

Worker-Owner, Sassafras Tech Collective
Specializing in innovative, usable tech for social change
sassafras.coop · @sassafrastech

Resident, Touchstone Cohousing
touchstonecohousing.org

Hi Caesar,

You are right, ODK-Collect doesn't have the folder /res/raw. You can create
the required folder and put your .bks file in it, and your self-signed
certificate should work. Let me know if you have any other issues.

Regards,
Raghu Mittal
Handheld Solutions & Research Labs Pvt Ltd
handsrel.com

··· On Thu, Oct 30, 2014 at 9:53 PM, Mitch Sundt wrote:

Note also that some organizations can get free commercial SSL certificates
(so you don't need tocode in a self-signed one):

FREE SSL certificates are available for charities and non-profits
https://www.comodo.com/news/press_releases/10_01_06.html

FREE SSL certificates are available for open-source projects (probably not
applicable to most people):
https://www.globalsign.eu/company/press/061913-globalsign-offers-free-ssl-certificates-for-open-source-projects.html

On Thu, Oct 30, 2014 at 5:53 AM, Tom Smyth tom@sassafras.coop wrote:

Caesar, what tools are you using to compile? ant? eclipse? Where is it
telling you you need /res/raw? I've never seen that before. Can you post
the compile command you're giving and what errors you're getting?

On 30 October 2014 08:49, Caesar Kennedy Olima caesar.olima@gmail.com wrote:

Hi Raghu,

Thanks for the tips you mentioned in the blogposts. I would like to
recompile an Androd app from *opendatakit *source but cannot find the
/res/raw/ inside the *androidapp *dir. I just forked the latest code
as at 30th Oct, 2014. Anything I am missing? Looking forward to your
response.

Regards,
Caesar

--
You received this message because you are subscribed to the Google
Groups "ODK Developers" group.
To unsubscribe from this group and stop receiving emails from it, send
an email to opendatakit-developers+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--
Tom Smyth

Worker-Owner, Sassafras Tech Collective
Specializing in innovative, usable tech for social change
sassafras.coop · @sassafrastech

Resident, Touchstone Cohousing
touchstonecohousing.org

--
You received this message because you are subscribed to the Google Groups
"ODK Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to opendatakit-developers+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--
Mitch Sundt
Software Engineer
University of Washington
mitchellsundt@gmail.com

--
You received this message because you are subscribed to a topic in the
Google Groups "ODK Developers" group.
To unsubscribe from this topic, visit
https://groups.google.com/d/topic/opendatakit-developers/UMPyRcgQq1k/unsubscribe
.
To unsubscribe from this group and all its topics, send an email to
opendatakit-developers+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Thanks Raghu,

I will go through the steps as you suggest and give you my feedback. Thanks
for the support.

Regards,
Caesar

Hello ODK Developers,

After some trial with the ODK Collect code and following the process of
using a self signed certicicate, I have't had much success in compiling the
code.

Basically, I cloned the code and loaded it to Eclipse as Git project.
After trying to compile the first time, I noticed that possibly I was
missing some files and this prompted me to add some jar files to the libs
folder (google-maps, google-play-services,google-play-services-lib,
com.google.maps.api13
, and android13).

On running the project, I get the an error from attached picture. I have
also attached an export from the logcat from eclipse. Has anyone gone
though a similar situation and might have a suggestion where I might have
gone wrong? All suggestions are greatly welcome.

Kind regards,
Caesar K. Olima

odk-collect-log.txt (5.24 KB)

Did you see
https://code.google.com/p/opendatakit/wiki/CollectSourceCodeOverview ?

You need to use that version of playservices.

Start simple.

First get the Android hello-world tutorial to work.

Then:

Create a new workspace directory (via your computer's file browser)
Clone the ODK Collect and playservices projects into subdirectories within
that directory.
Start Eclipse on this new workspace directory.
Configure Eclipse Preferences to use Android Google APIS for Android 4.2 or
higher.
Import existing projects into workspace, choosing both ODK Collect and
playservices.

This should build without errors.

Then try to run this on a Android 4.2 device or emulator. In either case,
there must
be an external SDCard configured for the device / emulator.

Once you do that, you can then rehost onto github or whatever.

··· On Tue, Nov 4, 2014 at 8:26 AM, Caesar K. Olima wrote:

Hello ODK Developers,

After some trial with the ODK Collect code and following the process of
using a self signed certicicate, I have't had much success in compiling the
code.

Basically, I cloned the code and loaded it to Eclipse as Git project.
After trying to compile the first time, I noticed that possibly I was
missing some files and this prompted me to add some jar files to the libs
folder (google-maps, google-play-services,google-play-services-lib,
com.google.maps.api13
, and android13).

On running the project, I get the an error from attached picture. I have
also attached an export from the logcat from eclipse. Has anyone gone
though a similar situation and might have a suggestion where I might have
gone wrong? All suggestions are greatly welcome.

Kind regards,
Caesar K. Olima

--
You received this message because you are subscribed to the Google Groups
"ODK Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to opendatakit-developers+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--
Mitch Sundt
Software Engineer
University of Washington
mitchellsundt@gmail.com

Hi Mitch,

Thanks a lot for the suggestions and compile recommendations. I went
through the the instructions from small app and was able to more clearly
get the flow. My Eclipse compiled code failed on the Emulator altogether
but as has been suggested from the forums here, an alternative was to try
out IntelliJ and that's what worked for me. I am able to follow on the
debug from the IntelliJ's console and Logcat better. Can't really tell why
it did not work in Eclipse but the solution in my case has been IntelliJ
and including *google-maps *and play-services jars as dependencies to the
ODK Collect source code, as suggested.

Also thanking all other community members that contributed so this learning
experience.

Best regards,
Caesar Olima.