Server configuration with reverse proxy returns timeout and proxy errors

Support
5

Finally, we managed to get it work. It was a tough one, actually there were two layers of problems. Unfortunately, since I wasn't the one who came with the solution, I couldn't write it down accurately. Sorry, future troubleshooter.

First, there was the problem with a certificate causing proxy errors. Re-issuing the certificate may have solved that.

Also, the part in Apache config file pasted above was thrown out.

Now we have a new file /etc/apache2/sites-available/odk-ssl.conf. I know little about it, but it is based on default-ssl.conf. The contents (without comments) are:

<IfModule mod_ssl.c>
SSLStaplingCache "shmcb:logs/ssl_stapling(32768)"
  <VirtualHost survey.example.com:443>
    ServerAdmin webmaster@localhost
    ServerName survey.example.com
    DocumentRoot /var/www/html
	
    Protocols h2 http/1.1
    Header always set Strict-Transport-Security "max-age=63072000"
	
    ErrorLog ${APACHE_LOG_DIR}/error.log
    CustomLog ${APACHE_LOG_DIR}/access.log combined
	
    SSLEngine on
	
    ProxyRequests   Off
    ProxyPass / http://127.0.0.1:4480/
    ProxyPassReverse / http://127.0.0.1:4480/
    
    <FilesMatch "\.(cgi|shtml|phtml|php)$">
      SSLOptions +StdEnvVars
    </FilesMatch>
    
    <Directory /usr/lib/cgi-bin>
      SSLOptions +StdEnvVars
    </Directory>
    
    Include /etc/letsencrypt/options-ssl-apache.conf
    SSLCertificateFile /etc/letsencrypt/live/survey.example.com/fullchain.pem
    SSLCertificateKeyFile /etc/letsencrypt/live/survey.example.com/privkey.pem
  </VirtualHost>
</IfModule>

This worked for me, it may or may not work for you. This file is also softlinked in the adjacent sites-enabled folder. Notice that separate certificate files for ODK are stored in /etc/letsencrypt/live/survey.example.com/.

After this the HTTP 500 Proxy Error went away. HTTPS was working, but now every database request timed out after 60 seconds and nothing happened. Enketo complained about a connection to Redis:

Error: Redis connection to enketo_redis_main:6379 failed - connect ETIMEDOUT 172.19.0.3:6379

The firewall was not the source of the problem. Finally, it was pinned down to the fact that something was broken. (No, sorry, we couldn't figure it out.)

What did help was simple:

docker-compose down
docker-compose up -d

Using docker-compose down is normally discouraged, as it causes detaching the database from the application. However, the database is not stored in a container, but in a separate entity called a volume and Docker does not delete volumes easily.*

It is a good idea to mark down the ID of the database volume before downing so you don't have to search for it.

Reattaching the database volume is actually easy.

docker-compose restart and off we go.

* Disclaimer: Of course you value your data and thus you did a backup because you know that things can go wrong, nothing is guaranteed and well-intended instructions from strangers on the internet may not be always right.

3 Likes