ODK Aggregate Docker image

Hello all,
I recently shared an ODK Aggregate Docker image to Docker Hub. Running the container requires a linked MySQL container, plus a handful of environment variables on the host machine to manage setup/configuration. Full details are available at the repo.

https://hub.docker.com/r/kharatsa/odkaggregate/

Thought I would share with the community, since the Aggregate server AWS setup instructions are a bit hairy. This should make the process of launching new ODK Aggregate servers (relatively) easy, given some basic familiarity with Docker.

This ODK Aggregate image is based on the official Tomcat image - https://hub.docker.com/_/tomcat/. The build process automates Tomcat setup, with the addition of Connector/J. When the container is first launched, the startup script (run.sh) updates the necessary ODK Aggregate configuration files from the required environment variables. In effect, this automates the setup process provided in the ODK Aggregate installer GUI.

Hope someone finds this useful. If anyone has any thoughts or suggestions, I would love to hear them.

Sean

Good to hear this Sean. This is the way to go.

I am keen to know if it is possible to substitute MySQL with Postgres,
where Aggregate and postgres reside in two different containers/pods. Has
anyone tried this?

Rgds,
Geoeng Systems Ltd.

··· On Thursday, February 4, 2016 at 9:16:16 PM UTC+3, seanh...@gmail.com wrote: > > Hello all, > I recently shared an ODK Aggregate Docker image to Docker Hub. Running the > container requires a linked MySQL container, plus a handful of environment > variables on the host machine to manage setup/configuration. Full details > are available at the repo. > > https://hub.docker.com/r/kharatsa/odkaggregate/ > https://github.com/Kharatsa/odkaggregate > > Thought I would share with the community, since the Aggregate server AWS > setup instructions are a bit hairy. This should make the process of > launching new ODK Aggregate servers (relatively) easy, given some basic > familiarity with Docker. > > This ODK Aggregate image is based on the official Tomcat image - > https://hub.docker.com/_/tomcat/. The build process automates Tomcat > setup, with the addition of Connector/J. When the container is first > launched, the startup script (run.sh) updates the necessary ODK Aggregate > configuration files from the required environment variables. In effect, > this automates the setup process provided in the ODK Aggregate installer > GUI. > > Hope someone finds this useful. If anyone has any thoughts or suggestions, > I would love to hear them. > > Sean

I'm also super stoked to hear this! I'm going to take a crack at putting a docker-compose spec together for this.

Quick question - building that .war seems to entail installing Eclipse Kepler, old versions of Google AWT and all kinds of cruft that isn't part of my standard dev setup. Is there a command-line way to build the WAR?

Thanks for this great work!
Dav

Well, I still may do some work to be able to build my own solution, but for now, I can get things done quite nicely with docker compose. Here's a simple docker-compose.yml that you could execute via HOSTIP=$(docker-machine ip) docker-compose up. I feel like I am finally living in the future that Docker promised... thanks again Sean! (and all the authors of Aggregate of course :). Sean, you're welcome to steal this and add it to your repo as an alternative to deploy.sh. The "insecure" passwords aren't horribly insecure, as there is no route to the DB from the internet. Obviously, using my name as the default admin user is also not ideal... but all of this can be dealt with via environment variables.

aggregate:
image: kharatsa/odkaggregate
ports:
- "80:8080"
links:
- odkdb
environment:
# DB_CONTAINER_NAME is supposed to have a default, but
# get an unbound variable in run.sh line 33
DB_CONTAINER_NAME: odkdb
MYSQL_DATABASE: odk
MYSQL_USER: odk
MYSQL_PASSWORD: insecure
ODK_HOSTNAME: ${HOSTIP}
ODK_ADMIN_USERNAME: dav

odkdb:
image: mysql:5.7
environment:
MYSQL_ROOT_PASSWORD: insecure
MYSQL_DATABASE: odk
MYSQL_USER: odk
MYSQL_PASSWORD: insecure

Yes, you should be able to do that. You would need a different Docker image
configured for Postgres.

··· On Thu, Feb 11, 2016 at 8:25 PM, geoeng wrote:

Good to hear this Sean. This is the way to go.

I am keen to know if it is possible to substitute MySQL with Postgres,
where Aggregate and postgres reside in two different containers/pods. Has
anyone tried this?

Rgds,
Geoeng Systems Ltd.

On Thursday, February 4, 2016 at 9:16:16 PM UTC+3, seanh...@gmail.com wrote:

Hello all,
I recently shared an ODK Aggregate Docker image to Docker Hub. Running
the container requires a linked MySQL container, plus a handful of
environment variables on the host machine to manage setup/configuration.
Full details are available at the repo.

https://hub.docker.com/r/kharatsa/odkaggregate/
https://github.com/Kharatsa/odkaggregate

Thought I would share with the community, since the Aggregate server AWS
setup instructions are a bit hairy. This should make the process of
launching new ODK Aggregate servers (relatively) easy, given some basic
familiarity with Docker.

This ODK Aggregate image is based on the official Tomcat image -
https://hub.docker.com/_/tomcat/. The build process automates Tomcat
setup, with the addition of Connector/J. When the container is first
launched, the startup script (run.sh) updates the necessary ODK Aggregate
configuration files from the required environment variables. In effect,
this automates the setup process provided in the ODK Aggregate installer
GUI.

Hope someone finds this useful. If anyone has any thoughts or
suggestions, I would love to hear them.

Sean

--
You received this message because you are subscribed to the Google Groups
"ODK Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to opendatakit-developers+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--
Mitch Sundt
Software Engineer
University of Washington
mitchellsundt@gmail.com

See CONFIGURE.TXT in root directory and there are maven instructions for
tomcat and mysql/posgres. Not sure how up to date they are. See "Full Maven
Development Environment Configuration"

Waylon

··· On Thu, Mar 3, 2016 at 4:55 PM, wrote:

I'm also super stoked to hear this! I'm going to take a crack at putting a
docker-compose spec together for this.

Quick question - building that .war seems to entail installing Eclipse
Kepler, old versions of Google AWT and all kinds of cruft that isn't part
of my standard dev setup. Is there a command-line way to build the WAR?

Thanks for this great work!
Dav

--
You received this message because you are subscribed to the Google Groups
"ODK Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to opendatakit-developers+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

I don't know of a way to build the WAR outside the GUI. That said, the WAR is bundled with the Docker image in this case. The configuration details filled by the GUI are placeholder/dummy values, and are overwritten with your own configuration details provided with the first "docker run" command.

··· On Thursday, March 3, 2016 at 8:14:17 PM UTC-5, davc...@berkeley.edu wrote: > I'm also super stoked to hear this! I'm going to take a crack at putting a docker-compose spec together for this. > > Quick question - building that .war seems to entail installing Eclipse Kepler, old versions of Google AWT and all kinds of cruft that isn't part of my standard dev setup. Is there a command-line way to build the WAR? > > Thanks for this great work! > Dav

So now I've put together what (I hope) is a reasonable config for a small
number of concurrent users (<20) running on an EC2 t2.micro server. I
welcome further input on this. One corner case is that I'm pretty sure the
containers potential memory footprint still adds to potentially more than
the available memory... even if just by a sliver.

Hope this is helpful!

D

··· On Fri, Mar 4, 2016 at 2:30 PM, wrote:

Well, I still may do some work to be able to build my own solution, but
for now, I can get things done quite nicely with docker compose. Here's a
simple docker-compose.yml that you could execute via
HOSTIP=$(docker-machine ip) docker-compose up. I feel like I am finally
living in the future that Docker promised... thanks again Sean! (and all
the authors of Aggregate of course :). Sean, you're welcome to steal this
and add it to your repo as an alternative to deploy.sh. The "insecure"
passwords aren't horribly insecure, as there is no route to the DB from
the internet. Obviously, using my name as the default admin user is also
not ideal... but all of this can be dealt with via environment variables.

aggregate:
image: kharatsa/odkaggregate
ports:
- "80:8080"
links:
- odkdb
environment:
# DB_CONTAINER_NAME is supposed to have a default, but
# get an unbound variable in run.sh line 33
DB_CONTAINER_NAME: odkdb
MYSQL_DATABASE: odk
MYSQL_USER: odk
MYSQL_PASSWORD: insecure
ODK_HOSTNAME: ${HOSTIP}
ODK_ADMIN_USERNAME: dav

odkdb:
image: mysql:5.7
environment:
MYSQL_ROOT_PASSWORD: insecure
MYSQL_DATABASE: odk
MYSQL_USER: odk
MYSQL_PASSWORD: insecure

--
You received this message because you are subscribed to a topic in the
Google Groups "ODK Developers" group.
To unsubscribe from this topic, visit
https://groups.google.com/d/topic/opendatakit-developers/sgiNI3imsuA/unsubscribe
.
To unsubscribe from this group and all its topics, send an email to
opendatakit-developers+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--
Dav Clark
Data Scientist
Berkeley D-Lab + BIDS
bead.glass
917-544-8408

While I'm taking this thread off-topic...

··· On Thursday, March 3, 2016 at 8:11:08 PM UTC-8, Waylon wrote: > See CONFIGURE.TXT in root directory and there are maven instructions for tomcat and mysql/posgres. Not sure how up to date they are. See "Full Maven Development Environment Configuration"

I did see those directions. I tried skipping the "install Eclipse" part, and got to where Maven wanted to compile the postgres stuff and failed... I was trying at the time to build for google app engine, and didn't want to bother with the postgres components.

But I guess you're saying that it's possible to build using maven from the command line. Is it possible to do this without installing Eclipse?

Cheers,
D

It is fragile to depend upon building the WAR from source, and a
significant pain to do so.

You should use the ODK Aggregate installer to build the WAR.

As has been said elsewhere, the installer does not alter any Windows
Registry or system configuration on the box on which it runs -- it just
generates the files that are placed in the ODK Aggregate directory.

And really, the only thing the installer does for the WAR image is build
the:

WEB-INF/lib/ODKAggregate-settings.jar

So you can take the WAR from any run of the ODK Aggregate installer, remove
that file, then do your own reconstruction of that file (e.g.,
security-settings.xml ) to tailor it to whatever deployment you need. See

··· On Wed, Mar 16, 2016 at 12:11 PM, Dav Clark wrote:

So now I've put together what (I hope) is a reasonable config for a small
number of concurrent users (<20) running on an EC2 t2.micro server. I
welcome further input on this. One corner case is that I'm pretty sure the
containers potential memory footprint still adds to potentially more than
the available memory... even if just by a sliver.

https://github.com/glass-bead-labs/ODK-composed

Hope this is helpful!

D

On Fri, Mar 4, 2016 at 2:30 PM, davclark@berkeley.edu wrote:

Well, I still may do some work to be able to build my own solution, but
for now, I can get things done quite nicely with docker compose. Here's a
simple docker-compose.yml that you could execute via
HOSTIP=$(docker-machine ip) docker-compose up. I feel like I am finally
living in the future that Docker promised... thanks again Sean! (and all
the authors of Aggregate of course :). Sean, you're welcome to steal this
and add it to your repo as an alternative to deploy.sh. The "insecure"
passwords aren't horribly insecure, as there is no route to the DB from
the internet. Obviously, using my name as the default admin user is also
not ideal... but all of this can be dealt with via environment variables.

aggregate:
image: kharatsa/odkaggregate
ports:
- "80:8080"
links:
- odkdb
environment:
# DB_CONTAINER_NAME is supposed to have a default, but
# get an unbound variable in run.sh line 33
DB_CONTAINER_NAME: odkdb
MYSQL_DATABASE: odk
MYSQL_USER: odk
MYSQL_PASSWORD: insecure
ODK_HOSTNAME: ${HOSTIP}
ODK_ADMIN_USERNAME: dav

odkdb:
image: mysql:5.7
environment:
MYSQL_ROOT_PASSWORD: insecure
MYSQL_DATABASE: odk
MYSQL_USER: odk
MYSQL_PASSWORD: insecure

--
You received this message because you are subscribed to a topic in the
Google Groups "ODK Developers" group.
To unsubscribe from this topic, visit
https://groups.google.com/d/topic/opendatakit-developers/sgiNI3imsuA/unsubscribe
.
To unsubscribe from this group and all its topics, send an email to
opendatakit-developers+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--
Dav Clark
Data Scientist
Berkeley D-Lab + BIDS
bead.glass
917-544-8408

--
You received this message because you are subscribed to the Google Groups
"ODK Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to opendatakit-developers+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--
Mitch Sundt
Software Engineer
University of Washington
mitchellsundt@gmail.com

It is fragile to depend upon building the WAR from source, and a
significant pain to do so.

You should use the ODK Aggregate installer to build the WAR.

As has been said elsewhere, the installer does not alter any Windows
Registry or system configuration on the box on which it runs -- it just
generates the files that are placed in the ODK Aggregate directory.

​I feel I've been around this barn once on the list, looking for a way to
build the JAR / WAR without installing a relatively bespoke Eclipse
environment. I wasn't thinking of the Installer as a way to get the JAR.

For me, the installer is brittle where it matters - dealing with whatever
crusty infrastructure due to multifactor / single-signon auth​ in my
academic environments. On the flipside, as long as the WAR works on one
specific version of some linux base container, Docker insulates me from
fragility there:
kharatsa/odkaggregate
​ ​
will likely keep working for a long time.
​ Importantly (for me) it's immediately available from the command line,
and it allows me to move forward with other things right away!

That said, I'm all for making the tomcat ODK aggregate container more
robust. Would the following be your suggested approach:

  1. Generate a JAR with the Aggregate installer
  2. Copy that into a docker build based on a standard tomcat container, and
    include more access or documentations of the options specified in the
    following?


​ ​

An ideal approach (from my perspective) would be to dispense with the
Installer, and incorporate that logic into something like the kharatsa
image. I guess your suggested approach is already doing that to some
extent, but for now, that's a bit beyond me (I'm not an enterprise Java guy
since 2006).

But anyway, I'm happy to contribute. And, if any folks are local in the
Berkeley area, I'd be happy to host a short hack session on this in BIDS!

Best,

··· On Fri, Mar 18, 2016 at 10:40 AM, Mitch Sundt wrote: -- Dav Clark Data Scientist Berkeley D-Lab + BIDS bead.glass 917-544-8408

No, what I propose:

Before you build your docker image, grab the latest ODK Aggregate installer.

Then run the installer outside of the docker image with garbage arguments
to generate an install directory with the .war and the SQL initialization
script.

Explode the .war file. A .war file is just a .zip file; the same is true
for .jar files -- they are also just .zip files.

Move the ODKAggregate-settings.jar out of the exploded .war directory.

Configure a Docker image with Tomcat, etc., placing the exploded .war
directory (minus the jar) into the tomcat webapps tree (when you use the
Tomcat web UI to upload a WAR file, this is what it is doing).

Then write a docker initialization script that reconstructs and copies the
ODKAggregate-settings.jar file back into that Tomcat webapps path after
adjusting the content of its files with the correct hostname, port,
database hostname, port, etc. as needed by your docker launch parameters,
as described by the deployment configuration link above. And restart the
tomcat server (so that this updated jar file can be picked up).

If you don't understand the deployment configuration info, run the
installer a couple of times, extract the settings jar, explode it, and run
diffs to see what changes -- only 2 files will be impacted for a given
database (MySQL / PostgreSQL) choice, jdbc.properties and
security.properties -- the others are never changed.

This creates a docker image that has everything set up to run except for
this one jar file. The initialization script configures that jar file with
passed-in parameters that specify the hostnames and ports to use. Depending
upon how you are running your database, you may need to manually run a
synthesized SQL initialization script on the database so that the spun-up
docker image can access the database.

··· On Fri, Mar 18, 2016 at 1:01 PM, Dav Clark wrote:

On Fri, Mar 18, 2016 at 10:40 AM, Mitch Sundt mitchellsundt@gmail.com wrote:

It is fragile to depend upon building the WAR from source, and a
significant pain to do so.

You should use the ODK Aggregate installer to build the WAR.

As has been said elsewhere, the installer does not alter any Windows
Registry or system configuration on the box on which it runs -- it just
generates the files that are placed in the ODK Aggregate directory.

​I feel I've been around this barn once on the list, looking for a way to
build the JAR / WAR without installing a relatively bespoke Eclipse
environment. I wasn't thinking of the Installer as a way to get the JAR.

For me, the installer is brittle where it matters - dealing with whatever
crusty infrastructure due to multifactor / single-signon auth​ in my
academic environments. On the flipside, as long as the WAR works on one
specific version of some linux base container, Docker insulates me from
fragility there:
kharatsa/odkaggregate
​ ​
will likely keep working for a long time.
​ Importantly (for me) it's immediately available from the command line,
and it allows me to move forward with other things right away!

That said, I'm all for making the tomcat ODK aggregate container more
robust. Would the following be your suggested approach:

  1. Generate a JAR with the Aggregate installer
  2. Copy that into a docker build based on a standard tomcat container, and
    include more access or documentations of the options specified in the
    following?

https://github.com/opendatakit/opendatakit/wiki/Aggregate-Deployment-Configuration
​ ​

An ideal approach (from my perspective) would be to dispense with the
Installer, and incorporate that logic into something like the kharatsa
image. I guess your suggested approach is already doing that to some
extent, but for now, that's a bit beyond me (I'm not an enterprise Java guy
since 2006).

But anyway, I'm happy to contribute. And, if any folks are local in the
Berkeley area, I'd be happy to host a short hack session on this in BIDS!

Best,

Dav Clark
Data Scientist
Berkeley D-Lab + BIDS
bead.glass
917-544-8408

--
You received this message because you are subscribed to the Google Groups
"ODK Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to opendatakit-developers+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--
Mitch Sundt
Software Engineer
University of Washington
mitchellsundt@gmail.com