Upgrade to Central v2023.2 fails at "Stopping postgres server"

Support
1

We are getting an pg_ctl: server does not shut down error when upgrading ODK Central from v1.1 to v2023.2. The error occurs when upgrading the database by running the command docker compose up postgres. It seems that the pg server doesn't shut down. If I then rerun docker compose up postgres I get ERROR: New data found, but upgrade not flagged as complete.

Is there any way to restart the database upgrade process?

Here are the last few lines leading up to the error message:

central-postgres-1  | Thu Sep 28 11:29:40 AM GMT 2023 [upgrade-postgres.sh] Updating extensions...
central-postgres-1  | You are now connected to database "odk" as user "odk".
central-postgres-1  | ALTER EXTENSION
central-postgres-1  | ALTER EXTENSION
central-postgres-1  | Thu Sep 28 11:29:40 AM GMT 2023 [upgrade-postgres.sh] Regenerating optimizer statistics...
central-postgres-1  | vacuumdb: processing database "odk": Generating minimal optimizer statistics (1 target)
central-postgres-1  | vacuumdb: processing database "postgres": Generating minimal optimizer statistics (1 target)
central-postgres-1  | vacuumdb: processing database "template1": Generating minimal optimizer statistics (1 target)
central-postgres-1  | vacuumdb: processing database "odk": Generating medium optimizer statistics (10 targets)
central-postgres-1  | vacuumdb: processing database "postgres": Generating medium optimizer statistics (10 targets)
central-postgres-1  | vacuumdb: processing database "template1": Generating medium optimizer statistics (10 targets)
central-postgres-1  | vacuumdb: processing database "odk": Generating default (full) optimizer statistics
central-postgres-1  | vacuumdb: processing database "postgres": Generating default (full) optimizer statistics
central-postgres-1  | vacuumdb: processing database "template1": Generating default (full) optimizer statistics
central-postgres-1  | Thu Sep 28 11:29:42 AM GMT 2023 [upgrade-postgres.sh] Stopping postgres server...
central-postgres-1  | waiting for server to shut down............................................................... failed
central-postgres-1  | pg_ctl: server does not shut down
central-postgres-1  | HINT: The "-m fast" option immediately disconnects sessions rather than
central-postgres-1  | waiting for session-initiated disconnection.
central-postgres-1 exited with code 1

I've also attached the full output from docker compose up postgres

Frank.

odk_upgrade.txt (19.8 KB)

2

Hi @sokolic, before going further, do you have a full backup of your system? If so, you might try restoring the backup and going through the upgrade process again.

The error message from PostgreSQL suggests there's an open connection to the database. Is it possible there is another process which is accessing the database? This could be from one of the ODK Central docker containers, or from elsewhere in your system, or externally.

You could check for other docker containers with docker ps. The normal upgrade steps should have stopped all ODK Central containers.

You can also check from inside PostgreSQL with SELECT * FROM pg_stat_activity;. You should be able to run this from outside the container with:

docker exec central-postgres-1 psql -U YOUR_USERNAME -W -d YOUR_DATABASE_NAME -c 'SELECT * FROM pg_stat_activity'

There is discussion of a similar-looking issue at I can't upgrade from v1.5.3.

3

Thanks @alxndrsn, I'll try going through the process again. I've got a full back so will do a restore and try again.