Central v1.3 not updating SSL certs

aankrah · November 3, 2021, 12:24pm

Hello all,
I encountered the same issue (SSL expiration) and followed the guide found here as shared by @issa (thanks BTW) to upgrade from version [{"client": (v1.0.0), "server": (v1.0.3)}] to v1.3.

Thing is, the certificate didn't renew with this process and while combing through this, I found @yanokwa's suggestion and followed it. I then found this in one log file

2021-11-02 17:13:25,891:DEBUG:certbot._internal.main:certbot version: 1.20.0
2021-11-02 17:13:25,891:DEBUG:certbot._internal.main:Location of certbot entry point: /usr/local/bin/certbot
2021-11-02 17:13:25,891:DEBUG:certbot._internal.main:Arguments: ['--nginx']
2021-11-02 17:13:25,891:DEBUG:certbot._internal.main:Discovered plugins: PluginsRegistry(PluginEntryPoint#manual,PluginEntryPoint#null,PluginEntryPoint#standalone,PluginEntryPoint#webroot)
2021-11-02 17:13:25,913:DEBUG:certbot._internal.log:Root logging level set at 30
2021-11-02 17:13:25,914:DEBUG:certbot._internal.plugins.selection:Requested authenticator nginx and installer nginx
2021-11-02 17:13:25,914:DEBUG:certbot._internal.plugins.selection:No candidate plugin
2021-11-02 17:13:25,914:DEBUG:certbot._internal.plugins.selection:Selected authenticator None and installer None

and this in another

2021-11-02 16:55:43,517:DEBUG:certbot._internal.display.obj:Notifying user:
Certbot failed to authenticate some domains (authenticator: webroot). The Certificate Authority reported these problems:
Domain: [redacted]
Type: connection
Detail: Fetching http://[redacted]/.well-known/acme-challenge/ZHhRFuV61Dd7pYw78ZnbddNe46o1DfY0D3BXBWHXMh8: Timeout during connect (likely firewall problem)

Hint: The Certificate Authority failed to download the temporary challenge files created by Certbot. Ensure that the listed domains serve their content from the provided --webroot-path/-w and that files created there can be downloaded from the internet.

2021-11-02 16:55:43,517:DEBUG:certbot._internal.error_handler:Encountered exception:
Traceback (most recent call last):
File "/usr/local/lib/python3.7/dist-packages/certbot/_internal/auth_handler.py", line 90, in handle_authorizations
self._poll_authorizations(authzrs, max_retries, best_effort)
File "/usr/local/lib/python3.7/dist-packages/certbot/_internal/auth_handler.py", line 178, in _poll_authorizations
raise errors.AuthorizationError('Some challenges have failed.')
certbot.errors.AuthorizationError: Some challenges have failed.

Any ideas that might help my case?

yanokwa · November 3, 2021, 4:33pm

Thanks for reporting this issue. It's a bug that we are aware of and the fix is at https://github.com/getodk/central/pull/253. We expect it'll be released in a patch to Central late this week. If you can't wait, you can apply the code in the link above to your install and rebuild.

aankrah · November 5, 2021, 8:14am

Seeing as I have no idea about the areas outlined in your fix, I suppose I'll leave it to the professionals to handle.

However, if any member of the community would be kind enough to school me on the subject, I have been called a fast learner here and there .

yanokwa · November 5, 2021, 5:01pm

This has been fixed in ODK Central v1.3.3.

aankrah · November 5, 2021, 8:39pm

Thanks a lot for the update, pun intended😜. I am currently rocking the latest and greatest versions yet -

24ee74e5f974a518aa1cc8b06e7addb3be6b4690 (v1.3.3-2-g24ee74e)
5cc6fd79d112ce36d6298c61bb8817689c4c323b client (v1.3.2)
1d1a3a59969e61383da74119e405e67778b7a170 server (v1.3.3)

Guess I'll have to wait for the auto renew to kick in and save the day. I'll be sure to mark your last as a solution immediately the CA certificate gets renewed. Thanks again.

aankrah · November 6, 2021, 5:05pm

Will I need to do something else to get the certificate renewed? I ask cos I'm close to hitting 24 hours and the site is still rocking the expired certificate thus making it inaccessible.
I'd greatly appreciate a nod in the right direction.
Thanks.

yanokwa · November 7, 2021, 7:05am

Try docker-compose build, then docker-compose stop, then docker-compose up -d. If it still doesn't work, look at docker-compose logs nginx for a hint as to what might have gone wrong.

aankrah · November 7, 2021, 10:45am

Thanks for the pointer. I followed your outlined procedure and got this -

Attaching to central_nginx_1
nginx_1 | writing a new nginx configuration file..
nginx_1 | starting nginx with certbot..
nginx_1 | Starting the Nginx service
nginx_1 | Starting the autorenewal service
nginx_1 | Starting certificate renewal process
nginx_1 | Requesting an RSA certificate for 'cashew-nfaso.com'
nginx_1 | Renewing an existing certificate for cashew-nfaso.com
nginx_1 |
nginx_1 | Certbot failed to authenticate some domains (authenticator: webroot). The Certificate Authority reported these problems:
nginx_1 | Domain: cashew-nfaso.com
nginx_1 | Type: connection
nginx_1 | Detail: Fetching http://cashew-nfaso.com/.well-known/acme-challenge/eHG5l079mB7jI8u1sfxUQRHe6EeZvRH1LJEXMdP9ehU: Timeout during connect (likely firewall problem)
nginx_1 |
nginx_1 | Hint: The Certificate Authority failed to download the temporary challenge files created by Certbot. Ensure that the listed domains serve their content from the provided --webroot-path/-w and that files created there can be downloaded from the internet.

Which happens to be the same thing logged when I tried to renew the certificate manually while running Central server v1.0.3. Am I still missing something?
PS I tried accessing the logged challenge URL from a remote device and the server itself and had no luck. But when I replaced the domain with localhost on the server, I got a 404. Normal behaviour?

yanokwa · November 7, 2021, 3:01pm

v1.3.3-2-g24ee74e in the version text suggests you've made changes to your Central install. What does git status and git diff say. Be careful about posting the latter because it may have sensitive credentials.

aankrah · November 7, 2021, 4:18pm

git status returns

On branch master
Your branch is up to date with 'origin/master'.

Unmerged paths:
  (use "git reset HEAD <file>..." to unstage)
  (use "git add/rm <file>..." as appropriate to mark resolution)

	deleted by us:   .env

no changes added to commit (use "git add" and/or "git commit -a")

git diff returns

* Unmerged path .env

During the initial update to v1.3.3, I got an error with git stash so I copied my .env file to a different location and brought it back after the git pull command.

yanokwa · November 7, 2021, 6:11pm

And what is in the .env file? It should look like this:

DOMAIN=central.example.com
SYSADMIN_EMAIL=example@server.com
SSL_TYPE=letsencrypt

aankrah · November 7, 2021, 6:25pm

It does. Only in a slightly different arrangement. I'm guessing that has no bearing, right?

yanokwa · November 8, 2021, 5:51pm

Not sure what is happening with your install, @aankrah. Do your certbot.conf and redirector.conf files look like this?

docker exec -it central_nginx_1 cat /etc/nginx/conf.d/certbot.conf

certbot.conf

server {
    # Listen only on port 81 for localhost, and nothing else.
    server_name 127.0.0.1;
    listen 127.0.0.1:81 default_server;

    charset utf-8;

    # Certbot's folder used for the ACME challenge response.
    location ^~ /.well-known/acme-challenge {
        default_type text/plain;
        root /var/www/letsencrypt;
    }
}

docker exec -it central_nginx_1 cat /etc/nginx/conf.d/redirector.conf

redirector.conf

server {
    # Listen on plain old HTTP and catch all requests so they can be redirected
    # to HTTPS instead.
    listen 80 default_server reuseport;
    listen [::]:80 default_server reuseport;

    # Pass this particular URL off to the certbot server so it can properly
    # respond to the Let's Encrypt ACME challenges for the HTTPS certificates.
    location '/.well-known/acme-challenge' {
        default_type "text/plain";
        proxy_pass http://localhost:81;
    }

    # Everything else gets shunted over to HTTPS for each user defined
    # server to handle.
    location / {
        return 301 https://$http_host$request_uri;
    }

aankrah · November 8, 2021, 6:02pm

@yanokwa I pretty much have exactly what you put in your post.

One thing to note though is, we only punched holes in our firewall for ports 80 and 443. Will we need to replicate for port 81 (as I saw it used in both the redirector and certbot configs) or is this resolved within docker to one of the other ports?

yanokwa · November 8, 2021, 6:07pm

I believe, but have not confirmed, that only port 80/443 necessary. https://crt.sh/?q=cashew-nfaso.com suggests that things have been working well for a while.

But if you have a firewall, then strange things might happen. I'd try turning that off and see if strange things stop happening.

aankrah · November 8, 2021, 6:11pm

It's strange the firewall might be doing this now (assuming that's the reason). I say this because it has been active since the initial install date and the CA certs have renewed just fine (as captured on https://crt.sh/?q=cashew-nfaso.com) till recently.

Will speak with a few colleague to see if we could pull it down long enough to stop and rebuild the server instance. Thanks a ton.

Timon · November 9, 2021, 1:46pm

Hi,

I have a similar issue. My certificate would not renew, so I upgraded to the latest version. But I still cannot access Central. I tried @yanokwa advice:

Try docker-compose build, then docker-compose stop, then docker-compose up -d. If it still doesn't work, look at docker-compose logs nginx for a hint as to what might have gone wrong.

and got similar results as @aankrah , only not with nginx_1 | Type: connection but with nginx_1 | Type: unauthorized.

My .env file looks like this:

DOMAIN=central.example.com
SYSADMIN_EMAIL=example@server.com
SSL_TYPE=letsencrypt

HTTP_PORT=80
HTTPS_PORT=443

and my certbot.conf and redirector.conf look the same as above.

aankrah · November 9, 2021, 2:15pm

Hello again community,
Today, I tried docker-compose build, docker-compose stop then docker-compose up -d all while in a DMZ (outside the firewall) and still got the same error I've been getting all along. I took things a step further this time and paid particular attention to the latter part of the nginx log.

nginx log

nginx_1 | Certbot failed to authenticate some domains (authenticator: webroot). The Certificate Authority reported these problems:
nginx_1 | Domain: cashew-nfaso.com
nginx_1 | Type: connection
nginx_1 | Detail: Fetching http://cashew-nfaso.com/.well-known/acme-challenge/nUa6RwXg7WQt_n0sUikYhKGCLcfeKFfPopXrx26uhwg: Timeout during connect (likely firewall problem)
nginx_1 |
nginx_1 | Hint: The Certificate Authority failed to download the temporary challenge files created by Certbot. Ensure that the listed domains serve their content from the provided --webroot-path/-w and that files created there can be downloaded from the internet.
nginx_1 |
nginx_1 | Saving debug log to /var/log/letsencrypt/letsencrypt.log
nginx_1 | Exiting abnormally:
nginx_1 | Traceback (most recent call last):
nginx_1 | File "/usr/local/bin/certbot", line 8, in
nginx_1 | sys.exit(main())
nginx_1 | File "/usr/local/lib/python3.7/dist-packages/certbot/main.py", line 15, in main
nginx_1 | return internal_main.main(cli_args)
nginx_1 | File "/usr/local/lib/python3.7/dist-packages/certbot/_internal/main.py", line 1574, in main
nginx_1 | return config.func(config, plugins)
nginx_1 | File "/usr/local/lib/python3.7/dist-packages/certbot/_internal/main.py", line 1434, in certonly
nginx_1 | lineage = _get_and_save_cert(le_client, config, domains, certname, lineage)
nginx_1 | File "/usr/local/lib/python3.7/dist-packages/certbot/_internal/main.py", line 122, in _get_and_save_cert
nginx_1 | renewal.renew_cert(config, domains, le_client, lineage)
nginx_1 | File "/usr/local/lib/python3.7/dist-packages/certbot/_internal/renewal.py", line 335, in renew_cert
nginx_1 | new_cert, new_chain, new_key, _ = le_client.obtain_certificate(domains, new_key)
nginx_1 | File "/usr/local/lib/python3.7/dist-packages/certbot/_internal/client.py", line 384, in obtain_certificate
nginx_1 | orderr = self._get_order_and_authorizations(csr.data, self.config.allow_subset_of_names)
nginx_1 | File "/usr/local/lib/python3.7/dist-packages/certbot/_internal/client.py", line 434, in _get_order_and_authorizations
nginx_1 | authzr = self.auth_handler.handle_authorizations(orderr, self.config, best_effort)
nginx_1 | File "/usr/local/lib/python3.7/dist-packages/certbot/_internal/auth_handler.py", line 90, in handle_authorizations
nginx_1 | self._poll_authorizations(authzrs, max_retries, best_effort)
nginx_1 | File "/usr/local/lib/python3.7/dist-packages/certbot/_internal/auth_handler.py", line 178, in _poll_authorizations
nginx_1 | raise errors.AuthorizationError('Some challenges have failed.')
nginx_1 | certbot.errors.AuthorizationError: Some challenges have failed.
nginx_1 | Ask for help or search for solutions at https://community.letsencrypt.org. See the logfile /var/log/letsencrypt/letsencrypt.log or re-run Certbot with -v for more details.
nginx_1 | Certbot failed for 'cashew-nfaso.com'. Check the logs for details.
nginx_1 | 2021/11/09 13:59:11 [notice] 204#204: signal process started
nginx_1 | Autorenewal service will now sleep 8d

I then run cat /var/log/letsencrypt/letsencrypt.log and found this

letsencrypt.log

2021-11-09 01:32:14,322:DEBUG:urllib3.connectionpool:http://localhost:None "GET /v2/connections?snap=certbot&interface=content HTTP/1.1" 200 97
2021-11-09 01:32:14,812:DEBUG:certbot._internal.main:certbot version: 1.21.0
2021-11-09 01:32:14,812:DEBUG:certbot._internal.main:Location of certbot entry point: /snap/certbot/1582/bin/certbot
2021-11-09 01:32:14,812:DEBUG:certbot._internal.main:Arguments: ['-q', '--preconfigured-renewal']
2021-11-09 01:32:14,813:DEBUG:certbot._internal.main:Discovered plugins: PluginsRegistry(PluginEntryPoint#apache,PluginEntryPoint#manual,PluginEntryPoint#nginx,PluginEntryPoint#null,PluginEntryPoint#standalone,PluginEntryPoint#webroot)
2021-11-09 01:32:14,827:DEBUG:certbot._internal.log:Root logging level set at 40
2021-11-09 01:32:14,831:DEBUG:certbot._internal.display.obj:Notifying user:

2021-11-09 01:32:14,831:DEBUG:certbot._internal.display.obj:Notifying user: No renewals were attempted.
2021-11-09 01:32:14,831:DEBUG:certbot._internal.display.obj:Notifying user: - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
2021-11-09 01:32:14,831:DEBUG:certbot._internal.renewal:no renewal failures

Hope this helps paint a better picture to someone.

yanokwa · November 9, 2021, 11:00pm

@aankrah @Timon There seems to be something else going on and I'm worried that other people might be affected. If you are willing to have us log into your server and take a look, please email support@getodk.org so we can coordinate that in a secure manner.

yanokwa · November 11, 2021, 4:57pm

I have reviewed both servers and I'm convinced that this is not a Central issue. Or rather, both servers have some upstream network filtering that is preventing access to port 80.