DB removed when upgrading ODK Central

Hello,

I recently updated to the latest ODK Central, following the instructions:
https://docs.getodk.org/central-install-digital-ocean/
It is not the first time that I upgrade, but, this time, the DB was removed (luckily, it was a test installation)

Is this expected?

Thank you!

Hi @alfonso and sorry you're having this trouble. Is there any chance that you might have done a docker-compose down instead of a docker-compose stop at any point? That would mean that a subsequent up would create a new database volume instead of using the existing one.

If that is what happened, it should be recoverable and I can provide instructions for that. We are looking into ways to migrate to a configuration that would be resilient to a down (a down should never be needed but users may already have that habit).

3 Likes

Hi @LN, indeed, I think you are right, at some point I did docker-compose down.

Instructions for recovery would be great, thanks!

These are steps I came up with when exploring this issue as part of the conversation at Data persistence when upgrading/downgrading Central. If others have better strategies, please chime in.

The first step is to identify the data directory that currently is being used by your system. You can do this with docker inspect --type container central_postgres_1 -f '{{(index .Mounts 0).Source}}'. You should get a folder in /var/lib/docker/volumes/ with a long, random name. Make a note of the long name. I'll use target as a stand-in for that name below.

Then you need to find the volume that was previously mounted (before the down) and holds your data. I did this by first doing docker volume ls to get all of the volumes the system knows about. You'll see several volumes with long random names. One of them is the one you identified above and another is the one you're looking for. These names correspond to folders in /var/lib/docker/volumes. I manually copied and pasted each name to the end of that folder and added _data to create the full paths and listed the contents of each: ls /var/lib/docker/volumes/195[...]15412/_data/. I repeated until I found a directory (other than target from above) with folders with the pg_ prefix (e.g. pg_clog). Make a note of the long name. I'll use source for it.

Now you know the volume that holds your data (source) and the volume that's currently being mounted (target). Your goal is to put the contents of source at target. I did:

cd
cd central
docker-compose stop
pushd /var/lib/docker/volumes
mv target target.bak
mv source target
popd 
docker-compose up -d

Go to your site in a browser and try to log in with an account that previously existed. If that doesn't immediately work, try doing another stop followed by up.

Once things are working as expected, you can remove the backup folder you made: rm target.bak

Do you have prior Docker experience and so are already used to removing containers? Or maybe you instinctively used down as an opposite of up? Any context you can give us into what happened would be helpful!

4 Likes

Thank you @LN !
The instructions were helpful, it seems that the everything works as before.

About the context: I was experimenting some customization of the enketo installation, and wanted to restart the container. I am a (little) bit familiar with docker, and indeed I a instinctively used down and up, rather than stop.

1 Like

Great to hear that worked for you!

That makes sense. Writing up some instructions for that process is on our (long) todo list and hopefully that would help as well. In the mean time, you should be able to do anything you want to without down and if that turns out not to be the case, do let us know.

Hi @LN , I ran into a similar error and have followed the guide till the command beginning with 'pushd' and am getting a command not found error.

Could you help with this, pardon me if this seems like a pretty simple question.

What operating system are you using that doesn't have pushd?

Regardless, you could do something like this...

cd
cd central
docker-compose stop
cd /var/lib/docker/volumes
mv FIRST postgres.data.bak
mv SECOND FIRST
cd /var/lib/docker/volumes
docker-compose up -d

Couldn't cd into it either, that's when I figured it's to do with permissions.
So I modified the permissions, ran the commands.
Tested and found everything was back to normal.
DB connected.
And I finally put back the permissions the way they were.

I must say though, the file command output 5 'ASCII text' (not just one as was insinuated there would be in the guide).

1 Like