Central Backup Restore

To start things off, I am running Central client (v1.0.0), server (v1.0.3) on Ubuntu 18.04.5 LTS

App users reported an error preventing them from submitting forms to central.
error
On the server, running df -h revealed the root partition was full.


Further investigation revealed that the /var/lib/docker/overlay2/ folder had taken upwards of 420GB as can be seen in the image above.

The traditional docker system prune and docker system prune -a could only salvage 210MB of storage which enabled app users to submit their data to the server. I however needed to find a way to release the over 400GB worth of storage that had been gobbled up. I came across this and a combination of find /var/lib/docker/containers/ -type f -name “*.log” -delete and docker-compose down && docker-compose up -d saw the reclamation of more than 390GB as shown below.
send 2
However, you guessed it, I ended up with a fresh install of central🤦‍♂️. I tried restoring a backup from 10 days ago following instructions from here but I keep getting this error. NB The backup file is almost 12GB

Any ideas how I could at least force the backup restore?

Hi Anthony,

There is a bug in Central where tmp files are not deleted. It will be fixed in the next release, but see Errors with sending forms to ODK Central because hard drive is full for a workaround that we shared a month ago.

In your attempt to resolve the tmp issue, you ran docker-compose down which is unfortunate because it will reset your database. We warn about the dangers of running this command at https://docs.getodk.org/central-install and provide recovery instructions at https://docs.getodk.org/central-troubleshooting/#troubleshooting-docker-compose-down. Try the recovery steps and report back with what you find.

As to the backup restore, that does seem like a bug and I'll ask the team to look into it. Before we force anything, have you successfully restored backups of less than 10GB before?

Yaw

1 Like

Hello @yanokwa,
I'll be sure to check out the recovery instructions in the next minute or so. Will revert with anything I find.

On the matter of backup restoration, this is my first time trying to restore a backup. So no, I have never even tried to restore a backup in the past.

:partying_face::partying_face::partying_face::partying_face:
Everything is back!
Thanks a ton!

One question though. Is there any overhead associated with the long list of instances displayed when you run ```docker volumes ls``?

1 Like