ODK aggregate docker image for PostgreSQL

ggalmazor · June 18, 2018, 10:06am

If I understand your Dockerfile correctly, you are using Aggregate with an external PostgreSQL database. If this is correct I think you could improve it by doing two things:

First, you could avoid having to run the install by directly using the Gradle build result. We're already doing similar stuff for the installer and packer tasks, and it should be "easy" to repurpose one of them for the Docker image. The Gradle task would take care of producing all the conf files for a specific Docker setup.
Second, you could consider not building a Docker image at all. You could use a stock tomcat:8-alpine image and mount a volume with the Aggregate build files into the container's /var/lib/tomcat8/webapp/ROOT path
- This would make your setup less portable but it would make much easier changing confs and would take care of any writing to filesystem needs.

markgarcia · June 19, 2018, 2:20am

@yanokwa No problem. So you're planning to officially support dockerized deployment?

@ggalmazor Yes, as I just want to use a single shared Postgres service/instance for both Aggregate and our own website.

I did try to use a gradle build stage (basing off https://hub.docker.com/_/gradle/) but I hit some snags and I was afraid of doing something wrong (as said I don't have prior experience with the stack). The installer looks to be the canonical way of deployment and is ready to use without me building anything so I just used it. But yes I do understand that doing a gradle build and tweaking the configuration there should be the goal.

What I really just need is a way to reconfigure Aggregate without reinstalling / rebuilding the WAR file. I think using environment variables is the most straightforward way to do this. I do realize that this will have far reaching implications on how Aggregate do and use configuration hence the questions I presented. I want to approach this first without changing anything in the Aggregate source code if possible (i.e. making a fork).

danbjoseph · July 9, 2018, 7:46pm

A docker deployment of Aggregate would be very helpful for the Red Cross, as I think we could then easily incorporate Aggregate into our POSM stack install.

yanokwa · July 9, 2018, 11:16pm

We now have improved docker support in Aggregate! You can see the docs here:

Give it a try and let us know what you think!

[Updated 2019-11-19] The correct link is this one: https://github.com/opendatakit/aggregate/tree/master/docs

nicolas · November 6, 2018, 2:15am

Hi everyone,
First of all very good initiative to have created odk aggregate docker image with postgresql. I am a new user of ODK and I'm a fan.
I am interested to use this image as part of my academic background on a raspberry pi.
I'm just starting with docker so I don't understand all the details to properly install the image with docker compose (https://github.com/opendatakit/aggregate/blob/master/docs/build-and-run-with-docker-compose.md)
On the SD card of the raspberry pi, i flashed the image of hypriot (http://blog.hypriot.com/downloads/), but then I'm lost, I don't know what are the other steps.
I don't understand how to use: Build the setup with ./gradle clean docker ComposeBuild -xtest -PwarMode = complete
Do you have a little tutorial to know all the steps, please?
Thank you in advance for your help.
Regards,

Xiphware · November 6, 2018, 2:21am

FYI since yer playing with ODK on a rpi, you might also be interested in @chrissyhroberts most excellent writeup on running briefcase on an rpi.

ggalmazor · November 6, 2018, 8:51am

Hi, @nicolas!

I love RaspberryPi & ODK projects

In one hand, you need to ensure your raspi is ready for running Docker containers:

Have you installed Docker? Did you try to run docker run hello-world?
Have you installed Docker Compose? You can check this by running docker-compose

On the other hand, you need to build the Docker Compose build. You can (and should) do this in another, more powerful computer. This is a rough step list of what you should do:

Install Java8
Download the Aggregate git repo
Run the command ./gradlew clean dockerComposeBuild -xtest -PwarMode=complete

Now you should copy the build/docker-compose into the raspi and run it. You should follow the instructions from the Quick Guide

nicolas · November 7, 2018, 12:54am

Hi,
Thanks for your help.
I first tested the ggalmazor solution:

docker, docker-compose are installed correctly
for java I install it as shown on http://wp.brodzinski.net/uncategorized/install-latest-java-8-raspbian/
downloading the Aggregate git repo in odk folder
Run the command ./gradlew clean dockerComposeBuild -xtest -PwarMode=complete but I have this error - Error: Could not find or load main class org.gradle.wrapper.GradleWrapperMain
Do you have an idea why the command does not work?
Thank you in advance,
Regards

ggalmazor · November 7, 2018, 12:09pm

Hi, @nicolas!

I see that the guide for installing Java8 you've linked is to install Java8 on the raspi, but you shouldn't do that. You should install Java8 in another, more powerful computer and build the Aggregate Docker Compose setup there.

nicolas · November 7, 2018, 2:52pm

Hi @ggalmazor,
When I run the command ./gradlew clean dockerComposeBuild -xtest -PwarMode=complete
I have this error:

Error: Could not find or load main class org.gradle.wrapper.GradleWrapperMain

Do you have an idea why the command does not work?
Thank you in advance,
Regards

ggalmazor · November 7, 2018, 8:24pm

Hi, @nicolas!

Did you install Git LFS before cloning the Aggregate repo? This guide explains how to get the code

nicolas · November 8, 2018, 2:30am

Hi @ggalmazor,
Thanks for your help, it was me, now the application installs correctly.
However, when I want to change the password of the users, I have this error:

JSON change-password request to http://192.168.1.30/ssl/user-manage-passwords failed

What needs to change to make it work, knowing that I don't plan to install SSL certificate and the IP address could change?

Regards,

ggalmazor · November 8, 2018, 10:32am

Hi, @nicolas

I'm glad you were able to build the Docker Compose setup

Did you make any change to the docker-compose.yml file? Could you copy it here so that I can see what setup you're running?

nicolas · November 8, 2018, 8:08pm

Hi @ggalmazor,
Here is the docker-compose.yml file that i use to run odk aggregate

github.com

getodk/aggregate/blob/master/docker-compose/docker-compose.yml

version: "3"
services:
  db:
    image: postgres:9.6-alpine
    ports:
      - 5432:5432
    volumes:
      - ./initdb:/docker-entrypoint-initdb.d
      - ./pgdata:/var/lib/postgresql/data

  wait_for_db:
    image: dadarek/wait-for-dependencies
    depends_on:
      - db
    command: db:5432

  tomcat8:
    depends_on:
      - wait_for_db
    image: tomcat:alpine

This file has been truncated. show original

First, on the SD card of the RaspberryPI, I flashed the image of hypriot - https://github.com/hypriot/image-builder-rpi/releases/download/v1.9.0/hypriotos-rpi-v1.9.0.img.zip
Regards,

ggalmazor · November 8, 2018, 8:28pm

Thanks for sharing that conf file, @nicolas!

The first strange thing I see is that your Aggregate instance should run on the 8080 port, but the URL in the error message from your previous comment doesn't use any port.

Could you copy here the contents of the security.properties file from your docker-compose setup? You can find it at the relative path: webapps/ROOT/WEB-INF/classes/security.properties

nicolas · November 8, 2018, 8:43pm

Thank you for your reply.
Here is the contents of the security.properties file:

#Thu Nov 07 18:08:31 CET 2018
security.server.realm.realmString=ODK Aggregate
security.server.channelType=ANY_CHANNEL
wink.handlersFactoryClass=org.opendatakit.aggregate.odktables.impl.api.wink.AppEngineHandlersFactory
security.server.superUserUsername=administrator
security.server.forceHttpsLinks=false
security.server.superUser=
security.server.port=80
security.server.securePort=443
security.help.about=auto-generated on 1970-01-01T00:00:00+0000 for ODK Aggregate
security.server.deviceAuthentication=digest
security.server.hostname=
security.server.secureChannelType=ANY_CHANNEL

Regards

ggalmazor · November 8, 2018, 9:51pm

Thanks! It looks like that might be wrong.

Stop your Docker Compose setup with docker-compose down and try changing the line that has the property security.server.port so that it looks like this:

security.server.port=8080

Then try running it again with docker-compose up -d

nicolas · November 9, 2018, 2:35pm

Hi @ggalmazor,
Great, it works fine, very well. Thank you very much for the help.
I have another question, is it possible to automatically start the docker-compose.yml file, when starting the raspberry pi?
I think, that it must be indicated in the file yml:

version: "3"
services:
db:
image: postgres:9.6-alpine
restart: unless-stopped
ports:
- 5432:5432
volumes:
- ./initdb:/docker-entrypoint-initdb.d
- ./pgdata:/var/lib/postgresql/data

wait_for_db:
image: dadarek/wait-for-dependencies
restart: unless-stopped
depends_on:
- db
command: db:5432

tomcat8:
depends_on:
- wait_for_db
image: tomcat:alpine
restart: always
ports:
- 8080:8080
volumes:
- ./webapps:/usr/local/tomcat/webapps

Regards,

ggalmazor · November 9, 2018, 3:00pm

I'm glad that it's working now

I think the restart option could be useful for some users. Would you care to document it on the Docker Compose instructions for Aggregate? You can use the edit (pencil icon) button to edit it in place

nicolas · November 20, 2018, 2:31am

Hi @ggalmazor,
No problem, I will edit the file in github to indicate how to do an automatic restart of the containers.
For information, I redid the procedure at the beginning, I have an error message:

Pulling db (postgres: 9.6-alpine) ...
9.6-alpine: Pulling from library / postgres
ERROR: no matching manifest for linux

Replacing in the file image: postgres: 9.6-alpine by image: postgres: 9.6 it works.
Is it normal?
Regards,