Central upgrade - postgresql problem

Support
1

I'm following the upgrade steps and when I try to start postgresql I am notified that the upgrade has not run yet.

I'm thinking that the new postgresql cluster has been been upgraded, but the data files have not yet been migrated.

How do I fix this?

many thank

root@fiano:/srv/central# docker compose up postgres
[+] Running 1/0
:heavy_check_mark: Container central-postgres-1 Created 0.0s
Attaching to central-postgres-1
central-postgres-1 | Tue 25 Jul 2023 07:02:34 AM GMT [upgrade-postgres.sh] Checking for existing upgrade marker file...
central-postgres-1 | Tue 25 Jul 2023 07:02:34 AM GMT [upgrade-postgres.sh] !!!
central-postgres-1 | Tue 25 Jul 2023 07:02:34 AM GMT [upgrade-postgres.sh] !!! ERROR: Deletion request file created, but upgrade has not yet run!
central-postgres-1 | Tue 25 Jul 2023 07:02:34 AM GMT [upgrade-postgres.sh] !!!
central-postgres-1 | Tue 25 Jul 2023 07:02:34 AM GMT [upgrade-postgres.sh] !!! Please email support@getodk.org for assistance.
central-postgres-1 | Tue 25 Jul 2023 07:02:34 AM GMT [upgrade-postgres.sh] !!!
central-postgres-1 exited with code 1

2

Try starting the service postgres14 (with the 14 on the end) instead of just postgres?

The upgrade makes new a new Postgres 14 container (likely called central-postgres14-1) and keeps the old Postgres 9.6 container around to copy data from one to the other.

It sounds like as you try to start up the old container, it wants to delete the data but realizes there is something wrong/doesn't think the data has been migrated so it is not letting the data get deleted.

I think the thing to do is figure out the state of the new container. Does it exist? Does it have data in it? What did the upgrade process look like for you? What happens when you try to start it with docker compose up postgres14?

Running the command you posted above docker compose up postgres is what should have triggered the creation of new container and data migration (see Upgrade your Database -- Step 7: Start the database upgrade and wait for the process to exit in the upgrade docs), and later the cleaning up of that data (Step 12.3).

Does this file ./files/postgres14/upgrade/upgrade-successful exist? (It should if the upgrade was successful, but the error you're seeing is probably because this file doesn't exist.)

Does this file ./files/postgres14/upgrade/delete-old-data exist? If so, I wonder if its existence is stopping the upgrade from happening?

1 Like
3

This was just sorted out by the wonderful support team. Upgrade was from an older version and the wrong instructions were followed, causing the delete-old-data to block the upgrade.

2 Likes
4

I have landed into the same error while upgrading from 1.5.1;
On docker compose up postgres, the following error pops; [upgrade-postgres.sh] !!! ERROR: Deletion request file created, but upgrade has not yet run!

And thus 'No such file or directory' pops on trying ls ./files/postgres14/upgrade/upgrade-successful.

What could be the cause or how do I overcome this error?

5

Are you able to take a full disk backup/snapshot? If you are, then take a snapshot so your data is safe, remove ~/central/files/postgres14/upgrade/delete-old-data and re-run the upgrade.

If you do not have a full disk backup/snapshot, email support@getodk.org.

6

Many thanks @yanokwa. I have taken the snapshot. But on ~/central/files/postgres14/upgrade/delete-old-data, it says 'Permission denied'

7

Try prepending sudo to the rm command you are using.

8

it says 'command not found'

9

Sorry @yanokwa , I realised that I had missed the rm command before the line -

So I have run;

rm files/postgres14/upgrade/delete-old-data from the central directory and it works. Am progressing now. Let me report here shortly

10

the process continues well and ./files/postgres14/upgrade/upgrade-successful is now returned. And docker compose up -d succeeds well. But on trying to log into the web interface for the recommended quick spot checks, first the central interface takes unusually long to show up - and does not log me in; reports error code 502 - something went wrong

11

The database is likely running a migration. Read the service logs to confirm. If it's taking a very long time, it might be RAM limited so increase the memory allocation.

12

Many thanks @yanokwa ,
I have increased physical memory. But it didn't succeed immediately;

In the logs;
On;
docker logs --tail 100 central-service-1 reported logs including;

Error: Cannot parse config file: '/usr/odk/config/local.json': SyntaxError: Expected property name or '}' in JSON at position 174

So I had go back to the .json template and follow steps for a customised mail server, It worked well and now UpToDate.

Thanks again

1 Like