Problem Updating to latest version

alexcorran · May 30, 2025, 3:11pm

I've taken on supporting our ODK system, and i'm having some issues updating to the latest version.

We're moving from v2024.1.0, after following the instructions, i'm getting a "error code 502" on the login page when signing in again.

postgres has been running at 100% for an extended period.

Does anyone have any suggestions of what to try next?

We've set --max_old_space_size=12288, so i don't think memory is an issue.

Here's the docker compose logs output after attempting to sign in:

mail-1  | using non-split configuration scheme from /etc/exim4/exim4.conf.template
mail-1  |     1 LOG: MAIN
mail-1  |     1   exim 4.96 daemon started: pid=1, -q15m, listening for SMTP on port 25 (IPv6 and IPv4)
mail-1  |   274 Exim version 4.96 uid=100 gid=102 pid=274 D=80001
mail-1  |   274 Berkeley DB: Berkeley DB 5.3.28: (September  9, 2013)
mail-1  |   274 Support for: crypteq iconv() IPv6 GnuTLS TLS_resume move_frozen_messages DANE DKIM DNSSEC Event I18N OCSP PIPECONNECT PRDR Queue_Ramp SOCKS SRS TCP_Fast_Open
mail-1  |   274 Lookups (built-in): lsearch wildlsearch nwildlsearch iplsearch cdb dbm dbmjz dbmnz dnsdb dsearch nis nis0 passwd
mail-1  |   274 Authenticators: cram_md5 external plaintext
mail-1  |   274 Routers: accept dnslookup ipliteral manualroute queryprogram redirect
mail-1  |   274 Transports: appendfile/maildir/mailstore autoreply lmtp pipe smtp
mail-1  |   274 Fixed never_users: 0
mail-1  |   274 Configure owner: 0:0
mail-1  |   274 Size of off_t: 8
mail-1  |   274 LOG: queue_run MAIN
mail-1  |   274   Start queue run: pid=274
mail-1  |   274 LOG: queue_run MAIN
mail-1  |   274   End queue run: pid=274
postgres14-1  | Fri May 30 02:18:03 PM GMT 2025 [start-postgres.sh] Starting postgres...
postgres14-1  | 
postgres14-1  | PostgreSQL Database directory appears to contain a database; Skipping initialization
postgres14-1  | 
postgres14-1  | 2025-05-30 14:18:03.557 UTC [1] LOG:  starting PostgreSQL 14.10 (Debian 14.10-1.pgdg120+1) on x86_64-pc-linux-gnu, compiled by gcc (Debian 12.2.0-14) 12.2.0, 64-bit
postgres14-1  | 2025-05-30 14:18:03.559 UTC [1] LOG:  listening on IPv4 address "0.0.0.0", port 5432
postgres14-1  | 2025-05-30 14:18:03.559 UTC [1] LOG:  listening on IPv6 address "::", port 5432
postgres14-1  | 2025-05-30 14:18:03.576 UTC [1] LOG:  listening on Unix socket "/var/run/postgresql/.s.PGSQL.5432"
postgres14-1  | 2025-05-30 14:18:03.591 UTC [28] LOG:  database system was shut down at 2025-05-30 14:15:56 UTC
postgres14-1  | 2025-05-30 14:18:03.605 UTC [1] LOG:  database system is ready to accept connections
postgres14-1  | 2025-05-30 14:18:04.338 UTC [35] LOG:  incomplete startup packet
postgres-1    | Fri May 30 02:18:03 PM GMT 2025 [upgrade-postgres.sh] Checking for existing upgrade marker file...
postgres-1    | Fri May 30 02:18:03 PM GMT 2025 [upgrade-postgres.sh] Upgrade has been run previously.
postgres-1    | Fri May 30 02:18:03 PM GMT 2025 [upgrade-postgres.sh] Complete.
enketo-1      | [start-enketo.sh] Checking secrets exist...
enketo-1      | [start-enketo.sh] Generating enketo configuration...
enketo-1      | [start-enketo.sh] Starting enketo...
enketo-1      | yarn workspace v1.22.22
enketo-1      | yarn run v1.22.22
enketo-1      | $ node .
enketo-1      | Worker 2 ready for duty at port 8005! (environment: production)
enketo-1      | Worker 1 ready for duty at port 8005! (environment: production)
service-1     | wait-for-it: waiting 15 seconds for postgres14:5432
service-1     | wait-for-it: postgres14:5432 is available after 0 seconds
service-1     | generating local service configuration..
service-1     | running migrations..
enketo_redis_cache-1  | 1:C 30 May 2025 14:18:03.669 # WARNING Memory overcommit must be enabled! Without it, a background save or replication may fail under low memory condition. Being disabled, it can also cause failures without low memory condition, see ************* To fix this issue add 'vm.overcommit_memory = 1' to /etc/sysctl.conf and then reboot or run the command 'sysctl vm.overcommit_memory=1' for this to take effect.
enketo_redis_cache-1  | 1:C 30 May 2025 14:18:03.669 * oO0OoO0OoO0Oo Redis is starting oO0OoO0OoO0Oo
enketo_redis_cache-1  | 1:C 30 May 2025 14:18:03.669 * Redis version=7.4.2, bits=64, commit=00000000, modified=0, pid=1, just started
enketo_redis_cache-1  | 1:C 30 May 2025 14:18:03.669 * Configuration loaded
enketo_redis_cache-1  | 1:M 30 May 2025 14:18:03.669 * monotonic clock: POSIX clock_gettime
enketo_redis_cache-1  | 1:M 30 May 2025 14:18:03.670 * Running mode=standalone, port=6380.
enketo_redis_cache-1  | 1:M 30 May 2025 14:18:03.671 * Server initialized
enketo_redis_cache-1  | 1:M 30 May 2025 14:18:03.671 * Loading RDB produced by version 7.2.4
enketo_redis_cache-1  | 1:M 30 May 2025 14:18:03.671 * RDB age 127 seconds
enketo_redis_cache-1  | 1:M 30 May 2025 14:18:03.671 * RDB memory usage when created 32.22 Mb
enketo_redis_cache-1  | 1:M 30 May 2025 14:18:03.784 * Done loading RDB, keys loaded: 3799, keys expired: 0.
enketo_redis_cache-1  | 1:M 30 May 2025 14:18:03.784 * DB loaded from disk: 0.114 seconds
enketo_redis_cache-1  | 1:M 30 May 2025 14:18:03.784 * Ready to accept connections tcp
enketo_redis_main-1   | 1:C 30 May 2025 14:18:03.778 # WARNING Memory overcommit must be enabled! Without it, a background save or replication may fail under low memory condition. Being disabled, it can also cause failures without low memory condition, see ************* To fix this issue add 'vm.overcommit_memory = 1' to /etc/sysctl.conf and then reboot or run the command 'sysctl vm.overcommit_memory=1' for this to take effect.
nginx-1               | writing client config...
nginx-1               | .....+.+........+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++*...............+......+..+...+....+........+...+....+...+..+......+.......+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++*.+..................+....+......+.....+.......+.................+....+..+....+.....+....+......+...+.....+.+......+...+..+.........+.......+..+...+....+...+...+............+........+...+......+...+...+....+...........+....+......+..+.............+.................+......+......+.+.....+....+.....+....+...........+....+......+...+.....+.......+......+............+............+...............+...+........+.........+.+.....+...+.........+.+..............+.+.....+.+........+.+......+.....+.+..+...............+...+..........+.........+...+...........+.......+..+.+......+...+..+....+.........+...........+......+...............+.........+.+........+...+....+.........+............+...+......+...+......+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
nginx-1               | ...+.........+......+...+....+...+...+.........+...+.........+........+................+...............+......+.....+.+...+..+...+......+.+..+.......+...+...+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++*.+........+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++*......+.+..+.......+.....+.+.................+...+.......+...+............+..+..........+...+.....+...+.......+...+............+.........+.....+...+......+.+......+...............+..+......+.........+.+.....+.+...+................................+...+.+..+.......+..+....+............+..+.+.....+.+.....+...+.......+......+.........+...+..+...+...+.............+.....................+...+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
nginx-1               | -----
nginx-1               | writing fresh nginx templates...
nginx-1               | starting nginx for custom ssl and self-signed certs...
nginx-1               | 2025/05/30 14:18:05 [notice] 1#1: using the "epoll" event method
nginx-1               | 2025/05/30 14:18:05 [notice] 1#1: nginx/1.27.4
nginx-1               | 2025/05/30 14:18:05 [notice] 1#1: built by gcc 12.2.0 (Debian 12.2.0-14) 
nginx-1               | 2025/05/30 14:18:05 [notice] 1#1: OS: Linux 5.4.0-216-generic
nginx-1               | 2025/05/30 14:18:05 [notice] 1#1: getrlimit(RLIMIT_NOFILE): 1048576:1048576
nginx-1               | 2025/05/30 14:18:05 [notice] 1#1: start worker processes
nginx-1               | 2025/05/30 14:18:05 [notice] 1#1: start worker process 15
nginx-1               | 2025/05/30 14:18:05 [notice] 1#1: start worker process 16
nginx-1               | **.**.**.** - - [30/May/2025:14:18:12 +0000] "GET / HTTP/1.1" 200 1116 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/136.0.0.0 Safari/537.36" "-"
nginx-1               | 2025/05/30 14:18:12 [error] 16#16: *1 connect() failed (111: Connection refused) while connecting to upstream, client: **.**.**.**, server: *************, request: "GET /v1/sessions/restore HTTP/1.1", upstream: "*************/v1/sessions/restore", host: "*************", referrer: "*************/"
nginx-1               | **.**.**.** - - [30/May/2025:14:18:12 +0000] "GET /v1/sessions/restore HTTP/1.1" 502 552 "*************/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/136.0.0.0 Safari/537.36" "-"
nginx-1               | **.**.**.** - - [30/May/2025:14:18:12 +0000] "GET /client-config.json HTTP/1.1" 200 27 "*************/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/136.0.0.0 Safari/537.36" "-"
nginx-1               | **.**.**.** - - [30/May/2025:14:18:27 +0000] "GET /version.txt HTTP/1.1" 200 209 "*************/login?next=/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/136.0.0.0 Safari/537.36" "-"
nginx-1               | **.**.**.** - - [30/May/2025:14:18:34 +0000] "POST /v1/sessions HTTP/1.1" 502 552 "*************/login?next=/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/136.0.0.0 Safari/537.36" "-"
nginx-1               | 2025/05/30 14:18:34 [error] 16#16: *4 connect() failed (111: Connection refused) while connecting to upstream, client: **.**.**.**, server: *************, request: "POST /v1/sessions HTTP/1.1", upstream: "*************/v1/sessions", host: "*************", referrer: "*************/login?next=/"
nginx-1               | 2025/05/30 14:18:36 [error] 16#16: *4 connect() failed (111: Connection refused) while connecting to upstream, client: **.**.**.**, server: *************, request: "POST /v1/sessions HTTP/1.1", upstream: "*************/v1/sessions", host: "*************", referrer: "*************/login?next=/"
nginx-1               | **.**.**.** - - [30/May/2025:14:18:36 +0000] "POST /v1/sessions HTTP/1.1" 502 552 "*************/login?next=/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/136.0.0.0 Safari/537.36" "-"
nginx-1               | 172.18.0.1 - - [30/May/2025:14:18:41 +0000] "POST /v1/sessions HTTP/1.1" 421 173 "-" "curl/7.68.0" "-"
pyxform-1             | [2025-05-30 14:18:03 +0000] [1] [INFO] Starting gunicorn 23.0.0
pyxform-1             | [2025-05-30 14:18:03 +0000] [1] [INFO] Listening at: http://0.0.0.0:80 (1)
pyxform-1             | [2025-05-30 14:18:03 +0000] [1] [INFO] Using worker: sync
pyxform-1             | [2025-05-30 14:18:03 +0000] [7] [INFO] Booting worker with pid: 7
pyxform-1             | [2025-05-30 14:18:03 +0000] [8] [INFO] Booting worker with pid: 8
pyxform-1             | [2025-05-30 14:18:03 +0000] [9] [INFO] Booting worker with pid: 9
pyxform-1             | [2025-05-30 14:18:03 +0000] [10] [INFO] Booting worker with pid: 10
pyxform-1             | [2025-05-30 14:18:03 +0000] [11] [INFO] Booting worker with pid: 11
enketo_redis_main-1   | 1:C 30 May 2025 14:18:03.778 * oO0OoO0OoO0Oo Redis is starting oO0OoO0OoO0Oo
enketo_redis_main-1   | 1:C 30 May 2025 14:18:03.778 * Redis version=7.4.2, bits=64, commit=00000000, modified=0, pid=1, just started
enketo_redis_main-1   | 1:C 30 May 2025 14:18:03.778 * Configuration loaded
enketo_redis_main-1   | 1:M 30 May 2025 14:18:03.779 * monotonic clock: POSIX clock_gettime
enketo_redis_main-1   | 1:M 30 May 2025 14:18:03.781 * Running mode=standalone, port=6379.
enketo_redis_main-1   | 1:M 30 May 2025 14:18:03.783 * Server initialized
enketo_redis_main-1   | 1:M 30 May 2025 14:18:03.783 * Loading RDB produced by version 7.2.4
enketo_redis_main-1   | 1:M 30 May 2025 14:18:03.783 * RDB age 127 seconds
enketo_redis_main-1   | 1:M 30 May 2025 14:18:03.783 * RDB memory usage when created 11.32 Mb
enketo_redis_main-1   | 1:M 30 May 2025 14:18:03.939 * Done loading RDB, keys loaded: 23823, keys expired: 0.
enketo_redis_main-1   | 1:M 30 May 2025 14:18:03.939 * DB loaded from disk: 0.157 seconds
enketo_redis_main-1   | 1:M 30 May 2025 14:18:03.939 * Ready to accept connections tcp

jniles · May 30, 2025, 7:46pm

Hi @alexcorran ,

Welcome to the ODK Community! Sorry to hear about the botched upgrade.

Your --max_old_space_size setting seems really high. That configuration should be lower than the amount of memory (RAM) on your machine. Is your ODK instance hosted somewhere with more than 12GBs of memory? It might be worth checking (free -h on Linux).

I'm not seeing anything in the logs that would indicate an issue that you're seeing. Could you try posting just the logs for nginx?

Best of luck!

jniles · June 1, 2025, 5:28pm

Thanks for posting more complete logs.

It looks like the issue is in this section:

nginx-1               | **.**.**.** - - [30/May/2025:14:18:12 +0000] "GET / HTTP/1.1" 200 1116 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/136.0.0.0 Safari/537.36" "-"
nginx-1               | 2025/05/30 14:18:12 [error] 16#16: *1 connect() failed (111: Connection refused) while connecting to upstream, client: **.**.**.**, server: *************, request: "GET /v1/sessions/restore HTTP/1.1", upstream: "*************/v1/sessions/restore", host: "*************", referrer: "*************/"
nginx-1               | **.**.**.** - - [30/May/2025:14:18:12 +0000] "GET /v1/sessions/restore HTTP/1.1" 502 552 "*************/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/136.0.0.0 Safari/537.36" "-"
nginx-1               | **.**.**.** - - [30/May/2025:14:18:12 +0000] "GET /client-config.json HTTP/1.1" 200 27 "*************/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/136.0.0.0 Safari/537.36" "-"
nginx-1               | **.**.**.** - - [30/May/2025:14:18:27 +0000] "GET /version.txt HTTP/1.1" 200 209 "*************/login?next=/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/136.0.0.0 Safari/537.36" "-"
nginx-1               | **.**.**.** - - [30/May/2025:14:18:34 +0000] "POST /v1/sessions HTTP/1.1" 502 552 "*************/login?next=/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/136.0.0.0 Safari/537.36" "-"
nginx-1               | 2025/05/30 14:18:34 [error] 16#16: *4 connect() failed (111: Connection refused) while connecting to upstream, client: **.**.**.**, server: *************, request: "POST /v1/sessions HTTP/1.1", upstream: "*************/v1/sessions", host: "*************", referrer: "*************/login?next=/"
nginx-1               | 2025/05/30 14:18:36 [error] 16#16: *4 connect() failed (111: Connection refused) while connecting to upstream, client: **.**.**.**, server: *************, request: "POST /v1/sessions HTTP/1.1", upstream: "*************/v1/sessions", host: "*************", referrer: "*************/login?next=/"

To my knowledge, this would indicate that the nginx isn't connecting to the upstream ODK server. If postgres is spinning at 100%, it may be that the node service is just waiting for it before booting.

Have you tried to stop and start the server?

$ docker compose stop
$ docker compose up -d

alexcorran · June 2, 2025, 12:12pm

Thanks for the response!
We're running it on our VMware cluster, and it's got 16Gb RAM assigned to it. i assume the previous managed overprovisioned it to be safe.
I've tried restarting the server, but to no effect.

Here's the nginx logs, though doesn't look like much more info

nginx-1  | writing client config... 
nginx-1  | ..+......+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++*..+......................+.....+...+.+.....+......+.+..+............+..................+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++*...+........+.........................+..+....+......+..+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
nginx-1  | .........+.....+...+.......+.....+.+..+.......+........+...+.........+.+...+..+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++*.+.+...+......+...+.....+...+.......+...........+.+...+...+..+..................+....+...........+....+...+..+....+..+...+....+..+.+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++*.+.........+.......+..+.+..+.......+.....+......+....+........+.........+.+...+....................+......+.........+.+.....+......+.......+...+...+...........+...+.+......+...+......+....................+.+......+..+......+.+...+...+............+..+......+...+.......+............+...............+...+.........+........+.+...........+.+........+.........+...+....+..............+....+............+...........+...+....+.....+...+...+......+.........+......+...+............+.+........+....+..+.+.................+...+....+........+...+............+.+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
nginx-1  | -----
nginx-1  | writing fresh nginx templates...
nginx-1  | starting nginx for custom ssl and self-signed certs...
nginx-1  | 2025/06/02 11:59:54 [notice] 1#1: using the "epoll" event method
nginx-1  | 2025/06/02 11:59:54 [notice] 1#1: nginx/1.27.4
nginx-1  | 2025/06/02 11:59:54 [notice] 1#1: built by gcc 12.2.0 (Debian 12.2.0-14)
nginx-1  | 2025/06/02 11:59:54 [notice] 1#1: OS: Linux 5.4.0-216-generic
nginx-1  | 2025/06/02 11:59:54 [notice] 1#1: getrlimit(RLIMIT_NOFILE): 1048576:1048576
nginx-1  | 2025/06/02 11:59:54 [notice] 1#1: start worker processes
nginx-1  | 2025/06/02 11:59:54 [notice] 1#1: start worker process 14
nginx-1  | 2025/06/02 11:59:54 [notice] 1#1: start worker process 15
nginx-1  | 2025/06/02 11:59:54 [notice] 1#1: start worker process 16
nginx-1  | 2025/06/02 11:59:54 [notice] 1#1: start worker process 17
nginx-1  | 2025/06/02 11:59:54 [notice] 1#1: start worker process 18
nginx-1  | 2025/06/02 11:59:54 [notice] 1#1: start worker process 19
nginx-1  | 2025/06/02 11:59:54 [notice] 1#1: start worker process 20
nginx-1  | 2025/06/02 11:59:54 [notice] 1#1: start worker process 21
nginx-1  | 172.18.0.1 - - [02/Jun/2025:12:00:06 +0000] "POST /v1/sessions HTTP/1.1" 421 173 "-" "curl/7.68.0" "-"
nginx-1  | ************* - - [02/Jun/2025:12:00:09 +0000] "GET / HTTP/1.1" 200 1116 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/136.0.0.0 Safari/537.36" "-"
nginx-1  | ************* - - [02/Jun/2025:12:00:09 +0000] "GET /assets/style-DzUCcHXw.css HTTP/1.1" 200 45990 "*************/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/136.0.0.0 Safari/537.36" "-"
nginx-1  | ************* - - [02/Jun/2025:12:00:09 +0000] "GET /assets/index-iJmUQqPm.js HTTP/1.1" 200 180853 "*************/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/136.0.0.0 Safari/537.36" "-"
nginx-1  | ************* - - [02/Jun/2025:12:00:09 +0000] "GET /assets/hover-cards-s-WRn4cS.js HTTP/1.1" 200 2452 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/136.0.0.0 Safari/537.36" "-"
nginx-1  | ************* - - [02/Jun/2025:12:00:09 +0000] "GET /assets/date-time-B-2wtrCX.js HTTP/1.1" 200 885 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/136.0.0.0 Safari/537.36" "-"
nginx-1  | ************* - - [02/Jun/2025:12:00:09 +0000] "GET /assets/dl-data-CXPdAJXr.js HTTP/1.1" 200 616 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/136.0.0.0 Safari/537.36" "-"
nginx-1  | ************* - - [02/Jun/2025:12:00:09 +0000] "GET /assets/datetime-jGCpXsxq.js HTTP/1.1" 200 22610 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/136.0.0.0 Safari/537.36" "-"
nginx-1  | ************* - - [02/Jun/2025:12:00:09 +0000] "GET /assets/string-DjC5jGOS.js HTTP/1.1" 200 1228 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/136.0.0.0 Safari/537.36" "-"
nginx-1  | ************* - - [02/Jun/2025:12:00:09 +0000] "GET /assets/review-state-Do89RE6G.js HTTP/1.1" 200 661 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/136.0.0.0 Safari/537.36" "-"
nginx-1  | ************* - - [02/Jun/2025:12:00:09 +0000] "GET /assets/popover-CJr_9LuC.js HTTP/1.1" 200 951 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/136.0.0.0 Safari/537.36" "-"
nginx-1  | ************* - - [02/Jun/2025:12:00:09 +0000] "GET /assets/review-state-C1KMRR0F.js HTTP/1.1" 200 340 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/136.0.0.0 Safari/537.36" "-"
nginx-1  | ************* - - [02/Jun/2025:12:00:09 +0000] "GET /assets/submission-CoViIpUO.js HTTP/1.1" 200 547 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/136.0.0.0 Safari/537.36" "-"
nginx-1  | 2025/06/02 12:00:09 [error] 18#18: *5 connect() failed (111: Connection refused) while connecting to upstream, client: *************, server: ************, request: "GET /v1/sessions/restore HTTP/1.1", upstream: "************/v1/sessions/restore", host: "************", referrer: "*************/"
nginx-1  | ************* - - [02/Jun/2025:12:00:09 +0000] "GET /v1/sessions/restore HTTP/1.1" 502 552 "*************/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/136.0.0.0 Safari/537.36" "-"
nginx-1  | ************* - - [02/Jun/2025:12:00:09 +0000] "GET /client-config.json HTTP/1.1" 200 27 "*************/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/136.0.0.0 Safari/537.36" "-"
nginx-1  | ************* - - [02/Jun/2025:12:00:09 +0000] "GET /assets/outdated-version-Bf5SicjT.js HTTP/1.1" 200 1657 "*************/assets/index-iJmUQqPm.js" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/136.0.0.0 Safari/537.36" "-"
nginx-1  | ************* - - [02/Jun/2025:12:00:09 +0000] "GET /assets/modal-1lFZCzpf.js HTTP/1.1" 200 3718 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/136.0.0.0 Safari/537.36" "-"
nginx-1  | ************* - - [02/Jun/2025:12:00:09 +0000] "GET /assets/introduction-CCf1mIEo.js HTTP/1.1" 200 2748 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/136.0.0.0 Safari/537.36" "-"
nginx-1  | ************* - - [02/Jun/2025:12:00:09 +0000] "GET /fonts/icomoon.ttf?ohpk4j HTTP/1.1" 200 17256 "*************/assets/style-DzUCcHXw.css" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/136.0.0.0 Safari/537.36" "-"
nginx-1  | ************* - - [02/Jun/2025:12:00:09 +0000] "GET /favicon.ico HTTP/1.1" 200 15086 "*************/login?next=/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/136.0.0.0 Safari/537.36" "-"
nginx-1  | ************* - - [02/Jun/2025:12:00:09 +0000] "GET /favicon.ico HTTP/1.1" 304 0 "*************/login?next=/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/136.0.0.0 Safari/537.36" "-"
nginx-1  | ************* - - [02/Jun/2025:12:00:12 +0000] "GET /version.txt HTTP/1.1" 200 196 "*************/login?next=/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/136.0.0.0 Safari/537.36" "-"
nginx-1  | ************* - - [02/Jun/2025:12:00:12 +0000] "GET /favicon.ico HTTP/1.1" 304 0 "*************/version.txt" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/136.0.0.0 Safari/537.36" "-"
nginx-1  | ************* - - [02/Jun/2025:12:00:15 +0000] "GET /version.txt HTTP/1.1" 200 196 "*************/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/136.0.0.0 Safari/537.36" "-"
nginx-1  | 2025/06/02 12:00:23 [error] 20#20: *7 connect() failed (111: Connection refused) while connecting to upstream, client: *************, server: ************, request: "POST /v1/sessions HTTP/1.1", upstream: "************/v1/sessions", host: "************", referrer: "*************/login?next=/"
nginx-1  | ************* - - [02/Jun/2025:12:00:23 +0000] "POST /v1/sessions HTTP/1.1" 502 552 "*************/login?next=/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/136.0.0.0 Safari/537.36" "-"
nginx-1  | ************* - - [02/Jun/2025:12:00:24 +0000] "GET /version.txt HTTP/1.1" 304 0 "*************/login?next=/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/136.0.0.0 Safari/537.36" "-"
nginx-1  | 2025/06/02 12:00:48 [notice] 1#1: signal 3 (SIGQUIT) received, shutting down
nginx-1  | 2025/06/02 12:00:48 [notice] 14#14: gracefully shutting down
nginx-1  | 2025/06/02 12:00:48 [notice] 14#14: exiting
nginx-1  | 2025/06/02 12:00:48 [notice] 14#14: exit
nginx-1  | 2025/06/02 12:00:48 [notice] 15#15: gracefully shutting down
nginx-1  | 2025/06/02 12:00:48 [notice] 19#19: gracefully shutting down
nginx-1  | 2025/06/02 12:00:48 [notice] 17#17: gracefully shutting down
nginx-1  | 2025/06/02 12:00:48 [notice] 16#16: gracefully shutting down
nginx-1  | 2025/06/02 12:00:48 [notice] 15#15: exiting
nginx-1  | 2025/06/02 12:00:48 [notice] 18#18: gracefully shutting down
nginx-1  | 2025/06/02 12:00:48 [notice] 19#19: exiting
nginx-1  | 2025/06/02 12:00:48 [notice] 17#17: exiting
nginx-1  | 2025/06/02 12:00:48 [notice] 16#16: exiting
nginx-1  | 2025/06/02 12:00:48 [notice] 15#15: exit
nginx-1  | 2025/06/02 12:00:48 [notice] 18#18: exiting
nginx-1  | 2025/06/02 12:00:48 [notice] 19#19: exit
nginx-1  | 2025/06/02 12:00:48 [notice] 18#18: exit
nginx-1  | 2025/06/02 12:00:48 [notice] 21#21: gracefully shutting down
nginx-1  | 2025/06/02 12:00:48 [notice] 20#20: gracefully shutting down
nginx-1  | 2025/06/02 12:00:48 [notice] 17#17: exit
nginx-1  | 2025/06/02 12:00:48 [notice] 21#21: exiting
nginx-1  | 2025/06/02 12:00:48 [notice] 16#16: exit
nginx-1  | 2025/06/02 12:00:48 [notice] 20#20: exiting
nginx-1  | 2025/06/02 12:00:48 [notice] 21#21: exit
nginx-1  | 2025/06/02 12:00:48 [notice] 20#20: exit
nginx-1  | 2025/06/02 12:00:48 [notice] 1#1: signal 17 (SIGCHLD) received from 15
nginx-1  | 2025/06/02 12:00:48 [notice] 1#1: worker process 15 exited with code 0
nginx-1  | 2025/06/02 12:00:48 [notice] 1#1: signal 29 (SIGIO) received
nginx-1  | 2025/06/02 12:00:48 [notice] 1#1: signal 17 (SIGCHLD) received from 16
nginx-1  | 2025/06/02 12:00:48 [notice] 1#1: worker process 16 exited with code 0
nginx-1  | 2025/06/02 12:00:48 [notice] 1#1: signal 29 (SIGIO) received
nginx-1  | 2025/06/02 12:00:48 [notice] 1#1: signal 17 (SIGCHLD) received from 14
nginx-1  | 2025/06/02 12:00:48 [notice] 1#1: worker process 14 exited with code 0
nginx-1  | 2025/06/02 12:00:48 [notice] 1#1: worker process 19 exited with code 0
nginx-1  | 2025/06/02 12:00:48 [notice] 1#1: worker process 21 exited with code 0
nginx-1  | 2025/06/02 12:00:48 [notice] 1#1: signal 29 (SIGIO) received
nginx-1  | 2025/06/02 12:00:48 [notice] 1#1: signal 17 (SIGCHLD) received from 21
nginx-1  | 2025/06/02 12:00:48 [notice] 1#1: signal 17 (SIGCHLD) received from 18
nginx-1  | 2025/06/02 12:00:48 [notice] 1#1: worker process 18 exited with code 0
nginx-1  | 2025/06/02 12:00:48 [notice] 1#1: worker process 20 exited with code 0
nginx-1  | 2025/06/02 12:00:48 [notice] 1#1: signal 29 (SIGIO) received
nginx-1  | 2025/06/02 12:00:48 [notice] 1#1: signal 17 (SIGCHLD) received from 17
nginx-1  | 2025/06/02 12:00:48 [notice] 1#1: worker process 17 exited with code 0
nginx-1  | 2025/06/02 12:00:48 [notice] 1#1: exit
nginx-1  | writing client config...
nginx-1  | ........+..................+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++*...+.+.....+....+.....+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++*....+...+.......+..............+....+...+..+...+............+.+............+........+.+.....+......+...+.+.....+.+...+..+...................+.....+....+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
nginx-1  | ...........+...+...+......+.....+....+......+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++*...+..............+.+.....+......+....+.....+.........+.+...+..+.............+...+..+...+.........+...+.+..+...+....+........+...+.+......+..+.............+.........+........+......+.+......+.....+............+.+......+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++*.+.........+.....+.+.....+...+.........................+..+...+.......+...+.........+.....+.........................+..+..........+.....+......+.......+.................+....+..+...+....+.....+.+....................+...............+.+.....+.........+.+.....+.+.....+...+............+...................+...+......+..+........................+...+.+..+...+.+..............+.+...+........+.......+..........................+.......+.........+...+...+...+...........+.+............+...+..+......+..........+..............+.......+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
nginx-1  | writing fresh nginx templates...
nginx-1  | -----
nginx-1  | starting nginx for custom ssl and self-signed certs...
nginx-1  | 2025/06/02 12:01:08 [notice] 1#1: using the "epoll" event method
nginx-1  | 2025/06/02 12:01:08 [notice] 1#1: nginx/1.27.4
nginx-1  | 2025/06/02 12:01:08 [notice] 1#1: built by gcc 12.2.0 (Debian 12.2.0-14)
nginx-1  | 2025/06/02 12:01:08 [notice] 1#1: OS: Linux 5.4.0-216-generic
nginx-1  | 2025/06/02 12:01:08 [notice] 1#1: getrlimit(RLIMIT_NOFILE): 1048576:1048576
nginx-1  | 2025/06/02 12:01:08 [notice] 1#1: start worker processes
nginx-1  | 2025/06/02 12:01:08 [notice] 1#1: start worker process 14
nginx-1  | 2025/06/02 12:01:08 [notice] 1#1: start worker process 15
nginx-1  | 2025/06/02 12:01:08 [notice] 1#1: start worker process 16
nginx-1  | 2025/06/02 12:01:08 [notice] 1#1: start worker process 17
nginx-1  | 2025/06/02 12:01:08 [notice] 1#1: start worker process 18
nginx-1  | 2025/06/02 12:01:08 [notice] 1#1: start worker process 19
nginx-1  | 2025/06/02 12:01:08 [notice] 1#1: start worker process 20
nginx-1  | 2025/06/02 12:01:08 [notice] 1#1: start worker process 21
nginx-1  | ************* - - [02/Jun/2025:12:01:11 +0000] "GET /login?next=/ HTTP/1.1" 200 1116 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/136.0.0.0 Safari/537.36" "-"
nginx-1  | 2025/06/02 12:01:11 [error] 14#14: *1 connect() failed (111: Connection refused) while connecting to upstream, client: *************, server: ************, request: "GET /v1/sessions/restore HTTP/1.1", upstream: "************/v1/sessions/restore", host: "************", referrer: "*************/login?next=/"
nginx-1  | ************* - - [02/Jun/2025:12:01:11 +0000] "GET /v1/sessions/restore HTTP/1.1" 502 552 "*************/login?next=/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/136.0.0.0 Safari/537.36" "-"
nginx-1  | ************* - - [02/Jun/2025:12:01:11 +0000] "GET /client-config.json HTTP/1.1" 200 27 "*************/login?next=/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/136.0.0.0 Safari/537.36" "-"
nginx-1  | ************* - - [02/Jun/2025:12:01:11 +0000] "GET /favicon.ico HTTP/1.1" 304 0 "*************/login?next=/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/136.0.0.0 Safari/537.36" "-"
nginx-1  | ************* - - [02/Jun/2025:12:01:17 +0000] "POST /v1/sessions HTTP/1.1" 502 552 "*************/login?next=/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/136.0.0.0 Safari/537.36" "-"
nginx-1  | 2025/06/02 12:01:17 [error] 14#14: *1 connect() failed (111: Connection refused) while connecting to upstream, client: *************, server: ************, request: "POST /v1/sessions HTTP/1.1", upstream: "************/v1/sessions", host: "************", referrer: "*************/login?next=/"
nginx-1  | ************* - - [02/Jun/2025:12:01:26 +0000] "GET /version.txt HTTP/1.1" 304 0 "*************/login?next=/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/136.0.0.0 Safari/537.36" "-"
nginx-1  | 172.18.0.1 - - [02/Jun/2025:12:02:06 +0000] "POST /v1/sessions HTTP/1.1" 421 173 "-" "curl/7.68.0" "-"
nginx-1  | ************* - - [02/Jun/2025:12:02:26 +0000] "GET /version.txt HTTP/1.1" 304 0 "*************/login?next=/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/136.0.0.0 Safari/537.36" "-"
nginx-1  | ************* - - [02/Jun/2025:12:03:27 +0000] "GET /version.txt HTTP/1.1" 304 0 "*************/login?next=/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/136.0.0.0 Safari/537.36" "-"

Here's the current version it's upgraded to if it helps:

versions:
5ea792b2cd52aa8f8f702a928cf2478d52185cf6 (v2025.1.2-1-g5ea792b)
 092e6548ea95a2a046f88dd6bb234e9fb877c676 client (v2025.1.2)
 4ea612178d7eb9ef191e8548ef5e70c2084fa8fc server (v2025.1.0)

alexcorran · June 4, 2025, 1:56pm

Hi All

After about a day and a half, postgres cpu returned to normal and I'm now able to sign into the webpage again.

Looking at vcenter, cpu usage dropped sometime between 11:30pm:06/03 and 12:30am:06/04.
In the postgres logs, there are some messages about checkpoints occurring too frequently just before that time:

postgres14-1          | Mon Jun  2 12:08:54 PM GMT 2025 [start-postgres.sh] Starting postgres... 
postgres-1            | Mon Jun  2 12:08:54 PM GMT 2025 [upgrade-postgres.sh] Checking for existing upgrade marker file...
postgres14-1          | 
postgres14-1          | PostgreSQL Database directory appears to contain a database; Skipping initialization
postgres14-1          | 
postgres-1            | Mon Jun  2 12:08:54 PM GMT 2025 [upgrade-postgres.sh] Upgrade has been run previously.
postgres14-1          | 2025-06-02 12:08:54.679 UTC [1] LOG:  starting PostgreSQL 14.10 (Debian 14.10-1.pgdg120+1) on x86_64-pc-linux-gnu, compiled by gcc (Debian 12.2.0-14) 12.2.0, 64-bit
postgres14-1          | 2025-06-02 12:08:54.708 UTC [1] LOG:  listening on IPv4 address "0.0.0.0", port 5432
postgres14-1          | 2025-06-02 12:08:54.708 UTC [1] LOG:  listening on IPv6 address "::", port 5432
postgres14-1          | 2025-06-02 12:08:54.714 UTC [1] LOG:  listening on Unix socket "/var/run/postgresql/.s.PGSQL.5432"
postgres14-1          | 2025-06-02 12:08:54.726 UTC [29] LOG:  database system was shut down at 2025-06-02 12:08:50 UTC
postgres14-1          | 2025-06-02 12:08:54.732 UTC [1] LOG:  database system is ready to accept connections
postgres14-1          | 2025-06-02 12:08:55.141 UTC [36] LOG:  incomplete startup packet
postgres14-1          | 2025-06-03 04:01:01.365 UTC [1] LOG:  received fast shutdown request
postgres14-1          | 2025-06-03 04:01:01.375 UTC [1] LOG:  aborting any active transactions
postgres14-1          | 2025-06-03 04:01:01.375 UTC [37] FATAL:  terminating connection due to administrator command
postgres14-1          | 2025-06-03 04:01:01.375 UTC [37] STATEMENT:  
postgres14-1          |       DELETE FROM client_audits
postgres14-1          |       WHERE "blobId" NOT IN (
postgres14-1          |         SELECT "blobId" FROM submission_attachments
postgres14-1          |         WHERE "blobId" IS NOT NULL
postgres14-1          |         AND "isClientAudit" IS true
postgres14-1          |       );
postgres14-1          |       
postgres14-1          | 2025-06-03 04:01:01.377 UTC [1] LOG:  background worker "logical replication launcher" (PID 35) exited with exit code 1
postgres14-1          | 2025-06-03 04:01:01.397 UTC [30] LOG:  shutting down
postgres14-1          | 2025-06-03 04:01:01.420 UTC [1] LOG:  database system is shut down
postgres-1            | Mon Jun  2 12:08:54 PM GMT 2025 [upgrade-postgres.sh] Complete.
postgres14-1          | Tue Jun  3 04:01:20 AM GMT 2025 [start-postgres.sh] Starting postgres...
postgres14-1          | 
postgres14-1          | PostgreSQL Database directory appears to contain a database; Skipping initialization
postgres14-1          | 
postgres14-1          | 2025-06-03 04:01:21.267 UTC [1] LOG:  starting PostgreSQL 14.10 (Debian 14.10-1.pgdg120+1) on x86_64-pc-linux-gnu, compiled by gcc (Debian 12.2.0-14) 12.2.0, 64-bit
postgres14-1          | 2025-06-03 04:01:21.267 UTC [1] LOG:  listening on IPv4 address "0.0.0.0", port 5432
postgres14-1          | 2025-06-03 04:01:21.267 UTC [1] LOG:  listening on IPv6 address "::", port 5432
postgres14-1          | 2025-06-03 04:01:21.273 UTC [1] LOG:  listening on Unix socket "/var/run/postgresql/.s.PGSQL.5432"
postgres14-1          | 2025-06-03 04:01:21.281 UTC [29] LOG:  database system was shut down at 2025-06-03 04:01:01 UTC
postgres14-1          | 2025-06-03 04:01:21.316 UTC [1] LOG:  database system is ready to accept connections
postgres14-1          | 2025-06-03 04:01:21.862 UTC [36] LOG:  incomplete startup packet
postgres14-1          | 2025-06-03 23:17:03.556 UTC [30] LOG:  checkpoints are occurring too frequently (26 seconds apart)
postgres14-1          | 2025-06-03 23:17:03.556 UTC [30] HINT:  Consider increasing the configuration parameter "max_wal_size".
postgres14-1          | 2025-06-03 23:17:15.046 UTC [30] LOG:  checkpoints are occurring too frequently (12 seconds apart)
postgres14-1          | 2025-06-03 23:17:15.046 UTC [30] HINT:  Consider increasing the configuration parameter "max_wal_size".
postgres14-1          | 2025-06-03 23:17:22.873 UTC [30] LOG:  checkpoints are occurring too frequently (7 seconds apart)
postgres14-1          | 2025-06-03 23:17:22.873 UTC [30] HINT:  Consider increasing the configuration parameter "max_wal_size".
postgres14-1          | 2025-06-03 23:17:30.801 UTC [30] LOG:  checkpoints are occurring too frequently (8 seconds apart)
postgres14-1          | 2025-06-03 23:17:30.801 UTC [30] HINT:  Consider increasing the configuration parameter "max_wal_size".
postgres14-1          | 2025-06-03 23:17:39.325 UTC [30] LOG:  checkpoints are occurring too frequently (9 seconds apart)
postgres14-1          | 2025-06-03 23:17:39.325 UTC [30] HINT:  Consider increasing the configuration parameter "max_wal_size".
postgres14-1          | 2025-06-03 23:17:47.357 UTC [30] LOG:  checkpoints are occurring too frequently (8 seconds apart)
postgres14-1          | 2025-06-03 23:17:47.357 UTC [30] HINT:  Consider increasing the configuration parameter "max_wal_size".
postgres14-1          | 2025-06-03 23:17:55.626 UTC [30] LOG:  checkpoints are occurring too frequently (8 seconds apart)
postgres14-1          | 2025-06-03 23:17:55.626 UTC [30] HINT:  Consider increasing the configuration parameter "max_wal_size".
postgres14-1          | 2025-06-03 23:18:03.644 UTC [30] LOG:  checkpoints are occurring too frequently (8 seconds apart)
postgres14-1          | 2025-06-03 23:18:03.644 UTC [30] HINT:  Consider increasing the configuration parameter "max_wal_size".
postgres14-1          | 2025-06-03 23:18:11.480 UTC [30] LOG:  checkpoints are occurring too frequently (8 seconds apart)
postgres14-1          | 2025-06-03 23:18:11.480 UTC [30] HINT:  Consider increasing the configuration parameter "max_wal_size".
postgres14-1          | 2025-06-03 23:18:19.583 UTC [30] LOG:  checkpoints are occurring too frequently (8 seconds apart)
postgres14-1          | 2025-06-03 23:18:19.583 UTC [30] HINT:  Consider increasing the configuration parameter "max_wal_size".
postgres14-1          | 2025-06-03 23:18:28.282 UTC [30] LOG:  checkpoints are occurring too frequently (9 seconds apart)
postgres14-1          | 2025-06-03 23:18:28.282 UTC [30] HINT:  Consider increasing the configuration parameter "max_wal_size".
postgres14-1          | 2025-06-04 04:01:02.047 UTC [1] LOG:  received fast shutdown request
postgres14-1          | 2025-06-04 04:01:02.057 UTC [1] LOG:  aborting any active transactions
postgres14-1          | 2025-06-04 04:01:02.058 UTC [1] LOG:  background worker "logical replication launcher" (PID 35) exited with exit code 1
postgres14-1          | 2025-06-04 04:01:02.063 UTC [30] LOG:  shutting down
postgres14-1          | 2025-06-04 04:01:02.111 UTC [1] LOG:  database system is shut down
postgres14-1          | Wed Jun  4 04:01:23 AM GMT 2025 [start-postgres.sh] Starting postgres...
postgres14-1          | 
postgres14-1          | PostgreSQL Database directory appears to contain a database; Skipping initialization
postgres14-1          | 
postgres14-1          | 2025-06-04 04:01:24.273 UTC [1] LOG:  starting PostgreSQL 14.10 (Debian 14.10-1.pgdg120+1) on x86_64-pc-linux-gnu, compiled by gcc (Debian 12.2.0-14) 12.2.0, 64-bit
postgres14-1          | 2025-06-04 04:01:24.274 UTC [1] LOG:  listening on IPv4 address "0.0.0.0", port 5432
postgres14-1          | 2025-06-04 04:01:24.274 UTC [1] LOG:  listening on IPv6 address "::", port 5432
postgres14-1          | 2025-06-04 04:01:24.280 UTC [1] LOG:  listening on Unix socket "/var/run/postgresql/.s.PGSQL.5432"
postgres14-1          | 2025-06-04 04:01:24.288 UTC [28] LOG:  database system was shut down at 2025-06-04 04:01:02 UTC
postgres14-1          | 2025-06-04 04:01:24.321 UTC [1] LOG:  database system is ready to accept connections
postgres14-1          | 2025-06-04 04:01:24.813 UTC [35] LOG:  incomplete startup packet

Does anyone have any suggestions on what's happening here?

This is all running on a clone of our production VM.
It would be difficult to have the production service down for a day and a half.

The postgres container seems to have a limit on how much memory it can use, is there a way to increase that limit? Perhaps to make whatever process is going on faster?

I should also mention the server uses about 270GB of storage, not sure that's a lot or a little for an ODK database.