ODK Central refuses login after upgrade/migration

1. What is the problem? Be very detailed.
I'm running an existing ODK Central instance on Rancher/K8s/Azure as per my write-up at Installing ODK central in microsoft azure cloud.
This setup still can't send emails through our mail server, see ticket. Other apps can use this server from inside the same K8s cluster/project, which indicates the problem l could be on the config side from ODK Central.

I need to migrate to a different K8s cluster, so my IT ops colleagues have

  • copied the db
  • copied the resource map (config.json/local.json)
  • set the same environment variables (hostname)
  • spun up the same Docker images I've built from ODK Central's front- and backend

2. What app or server are you using and on what device and operating system? Include version numbers.
Service: https://hub.docker.com/r/dbcawa/odk_service 0.6.3 (built today from master)
Nginx: https://hub.docker.com/r/dbcawa/odk_nginx 0.6.3 (built today from master)

3. What you have you tried to fix the problem?

  • Api specs say "for security reasons, if login fails, we won't say what failed". Fair.
  • Currently trying to stand up new Central instance from scratch with copy from old db, then with empty db. Assuming central will migrate db as needed.

4. What steps can we take to reproduce the problem?

5. Anything else we should know or have? If you have a test form or screenshots or logs, attach below.

  • Logs from frontend show (for failed logins):
2019/09/19 05:27:15 [error] 12#12: *14 connect() failed (113: No route to host) while connecting to upstream, client:, server: , request: "POST /v1/sessions HTTP/1.1", upstream: "", host: "odkcentral-migrated.dbca.wa.gov.au", referrer: "https://odkcentral-migrated.dbca.wa.gov.au/" - - [19/Sep/2019:05:27:15 +0000] "POST /v1/sessions HTTP/1.1" 502 166 "https://odkcentral-migrated.dbca.wa.gov.au/" "Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:69.0) Gecko/20100101 Firefox/69.0"

2019/09/19 05:39:11 [error] 12#12: *16 connect() failed (113: No route to host) while connecting to upstream, client:, server: , request: "GET /v1/projects/ HTTP/1.1", upstream: "", host: "odkcentral-migrated.dbca.wa.gov.au"

So the route to the backend seems problematic.
In a shell attached to the frontend (nginx) I can ping the backend's IP and get a reply, so there is a route!

apt install inetutils-ping
reply from host (works)

On my old, working instance, we can see the backend running on 8383:

root@nginx-dd7bdcbff-dljz8:/# curl
{"message":"Expected an API version (eg /v1) at the start of the request URL.","code":404.2}root@nginx-dd7bdcbff-dljz8:/# 

root@service-8675c4bd4c-s8j69:/usr/odk# netstat -pln
Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address           Foreign Address         State       PID/Program name    
tcp6       0      0 :::8383                 :::*                    LISTEN      44/node             
Active UNIX domain sockets (only servers)
Proto RefCnt Flags       Type       State         I-Node   PID/Program name     Path
unix  2      [ ACC ]     STREAM     LISTENING     70976    33/node              /usr/odk/naught.ipc

On the new instance, nothing is running on 8383"

root@service-5d55b4bc59-qzt5h:/usr/odk# apt update
root@service-5d55b4bc59-qzt5h:/usr/odk# apt install net-tools
root@service-5d55b4bc59-qzt5h:/usr/odk# netstat -plant
Active Internet connections (servers and established)
Proto Recv-Q Send-Q Local Address           Foreign Address         State       PID/Program name    
tcp        0      0       TIME_WAIT   -                   
tcp        0      0       TIME_WAIT   -                   
tcp        0      0       TIME_WAIT   -                   
tcp        0      0       TIME_WAIT   -                   

root@service-5d55b4bc59-qzt5h:/usr/odk# ./start-odk.sh 
running migrations..
Unhandled rejection error: SSL connection is required. Please specify SSL options and retry.
    at Connection.parseE (/usr/odk/node_modules/pg/lib/connection.js:545:11)
    at Connection.parseMessage (/usr/odk/node_modules/pg/lib/connection.js:370:19)
    at Socket.<anonymous> (/usr/odk/node_modules/pg/lib/connection.js:113:22)
    at Socket.emit (events.js:203:13)
    at addChunk (_stream_readable.js:294:12)
    at readableAddChunk (_stream_readable.js:275:11)
    at Socket.Readable.push (_stream_readable.js:210:10)
    at TCP.onStreamRead (internal/stream_base_commons.js:166:17)

starting cron..
starting server.

So, where can I specify the SSL options between backend service and db?

We found this: https://stackoverflow.com/questions/22301722/ssl-for-postgresql-connection-nodejs

hi. i've read the post a couple of times, and it's unclear to me what exactly the problem actually is? is it an email issue, or is your backend service not running as expected, or are you having issues logging in?

are you using your own postgres instance rather than the compose-managed instance, and that connection requires SSL/TLS? the connection to postgres is normally plaintext (as is the connection from nginx to the backend service; nginx does SSL termination for us).

Same here :wink: birthing pains of translating something that works perfectly fine (compose) into a different environment (k8s with db in separate, dedicated db pod).

Let me try local db again. Above error with hosted db (as preferred by our IT) reads like db wants SSL but Central doesn't support SSL to db. Is that correct? Could Central support SSL to db?

My other difficulty is to understand the certificate/email setup, as again we have an existing infrastructure (ssl certs, nginx) and need to rip the self-contained lid off Central to shoe-horn it into our infra.

At the far end of this exercise could be a "set up Central on Kubernetes" option.

i'm not sure why Central would have trouble connecting to a database over SSL. if it's a matter of providing/plumbing some flags then i don't have a problem supporting it, but in general while i'm happy to help investigate and find solutions, we only officially support the bundle as we provide it. i just don't have the time and resources to support two dozen highly complex customized configurations.

the email component is the most flexible, since all central really wants is a SMTP relay of any kind. it's not choosy at all.

1 Like

FYI this happens when I try to login as a known user (confirmed existing on odk-cmd) to a copy of ODK Central talking to a copy of my working main ODK Central db.
No action required, just FYI.

My next steps are to rebuild the Central docker images after the proxy timeout patch has been merged (hesitant to fiddle with my working production instance), and use a fresh local db pod. That should take both "stale sessions" and "SSL to db" out of the picture.

Closing this issue up:
Installing ODK central in microsoft azure cloud with latest ODK Central master works now. Currently using a local db container as per vanilla setup and a local SMTP server.
The issues I had were likely resolved through rebuilding and restarting both front- and back-end containers.

Much <3 to y'all for support and patience.

1 Like