Nginx letsencrypt error after odk central upgrade

1. What is the issue? Please be detailed.

I have a digital ocean server (ubuntu) running docker, installed using the instruction as provided here, except it was install about 2 years ago, at server version 1.1.1.

I am attempting to upgrade the server to v2023.2, in order to make use of the form audit files feature. I've tried to carefully follow the steps here. Before upgrading, I backed up the .env file, and then restored it after the upgrade process. I followed the steps listed for upgrading to v2023.2 regarding the postgres database update.

After the upgrade process, the central-nginx process won't start:

CONTAINER ID   IMAGE                                 COMMAND                  CREATED          STATUS                          PORTS      NAMES
187683c5ea13   central-nginx                         "/bin/bash /scripts/…"   10 minutes ago   Restarting (1) 43 seconds ago              central-nginx-1

The log indicates a problem around letsencrypt keys:

writing fresh nginx templates...
starting nginx for letsencrypt...
2023/03/20 19:49:52 [warning] Could not find keyfile file '/etc/letsencrypt/live//privkey.pem' in '/etc/nginx/conf.d/odk.conf.nokey'
2023/03/20 19:49:52 [warning] Could not find fullchain file '/etc/letsencrypt/live//fullchain.pem' in '/etc/nginx/conf.d/odk.conf.nokey'
2023/03/20 19:49:52 [warning] Could not find chain file '/etc/letsencrypt/live//fullchain.pem' in '/etc/nginx/conf.d/odk.conf.nokey'
2023/03/20 19:49:52 [info] Starting the Nginx service
2023/03/20 19:49:52 [info] Running the autorenewal service
2023/03/20 19:49:52 [emerg] 108#108: a duplicate default server for 0.0.0.0:80 in /etc/nginx/conf.d/redirector.conf:4
nginx: [emerg] a duplicate default server for 0.0.0.0:80 in /etc/nginx/conf.d/redirector.conf:4

It was a while ago that I first created this server, but I believe it automatically set up ssl certificates via letsencrypt. I needed to point my domain name to the digital ocean droplet.

2. What steps can we take to reproduce this issue?
Could try to upgrade from 1.1.1 to 2023.2

3. What have you tried to fix the issue?

Checked that DOMAIN and SSL_TYPE are set in .env as per described in .env.template.

Rebooted the server.

4. Upload any forms or screenshots you can share publicly below.

It looks like DOMAIN isn't set correctly or maybe there is some whitespace or special character. Can you copy and paste the DOMAIN line from .env and post it here? It should look like this...

DOMAIN=central.example.com
SYSADMIN_EMAIL=support@example.com
SSL_TYPE=letsencrypt
...
1 Like

I feel a bit unsure about posting the domain and email on this forum, but here is the basic format. I copied the domain name to here to try to check for special characters, but it seems fine.

SSL_TYPE=letsencrypt
DOMAIN=XXX.me
SYSADMIN_EMAIL=XXX@gmail.com

HTTP_PORT=80
HTTPS_PORT=443

I also tried pinging the server using the domain name from my PC and it works.

Did you remove the domain name from the log you pasted? Based on your .env, I would expect to see:

Could not find keyfile file '/etc/letsencrypt/live/XXX.me/privkey.pem'

No the log file is exactly as it was.

I know the .me extension isn't common, I don't know if that could make any issues.

It occurred to me I may have only put the .env file in place after the docker compose build command, and before the docker compose up -d command. I'll try running the build command again when I'm able to.

It’s not the .me domain. You have to run docker compose build after adding the env file.

I restored the DO droplet to a pre-upgrade snapshot and went through the process again. It still hasn't worked as the central-nginx container is still repeatedly restarting, but the error log is slightly different:

writing fresh nginx templates...
starting nginx for letsencrypt...
2023/03/21 20:16:32 [info] Starting the Nginx service
2023/03/21 20:16:32 [emerg] 76#76: a duplicate default server for 0.0.0.0:80 in /etc/nginx/conf.d/redirector.conf:4
nginx: [emerg] a duplicate default server for 0.0.0.0:80 in /etc/nginx/conf.d/redirector.conf:4

The /etc/nginx/conf.d directory has 3 .conf files, certbot.conf, odk.conf and redirector.conf. certbot.conf has the following contents:

server {
    # Listen on plain old HTTP
    listen 80 default_server;
    listen [::]:80 default_server;

    # Pass this particular URL off to certbot, to authenticate HTTPS certificates
    location '/.well-known/acme-challenge' {
        default_type "text/plain";
        proxy_pass http://localhost:1337;
    }

    # Everything else gets shunted over to HTTPS
    location / {
        return 301 https://$http_host$request_uri;
    }
}

redirector.conf has the following contents:

server {
    # Listen on plain old HTTP and catch all requests so they can be redirected
    # to HTTPS instead.
    listen 80 default_server reuseport;
    listen [::]:80 default_server reuseport;

    # Anything requesting this particular URL should be served content from
    # Certbot's folder so the HTTP-01 ACME challenges can be completed for the
    # HTTPS certificates.
    location '/.well-known/acme-challenge' {
        default_type "text/plain";
        root /var/www/letsencrypt;
    }

    # Everything else gets shunted over to HTTPS for each user defined
    # server to handle.
    location / {
        return 301 https://$http_host$request_uri;
    }
}

odk.conf has a listen line for port 443.

Thanks for the help

Try deleting the nginx container and rebuilding.

docker compose stop;
docker compose rm nginx -v;
docker compose build;
docker compose up -d;

It worked! Thanks. These messages are in the log, but don't seem to be causing any issue:

2023/03/22 05:05:44 [warning] Could not find keyfile file '/etc/letsencrypt/live/XXX.me/privkey.pem' in '/etc/nginx/conf.d/odk.conf'
2023/03/22 05:05:44 [warning] Could not find fullchain file '/etc/letsencrypt/live/XXX.me/fullchain.pem' in '/etc/nginx/conf.d/odk.conf'
2023/03/22 05:05:44 [warning] Could not find chain file '/etc/letsencrypt/live/XXX.me/fullchain.pem' in '/etc/nginx/conf.d/odk.conf'
2023/03/22 05:05:44 [error] Important file(s) for '/etc/nginx/conf.d/odk.conf' are missing, disabling...