Central Backup Restore

To start things off, I am running Central client (v1.0.0), server (v1.0.3) on Ubuntu 18.04.5 LTS

App users reported an error preventing them from submitting forms to central.
error
On the server, running df -h revealed the root partition was full.


Further investigation revealed that the /var/lib/docker/overlay2/ folder had taken upwards of 420GB as can be seen in the image above.

The traditional docker system prune and docker system prune -a could only salvage 210MB of storage which enabled app users to submit their data to the server. I however needed to find a way to release the over 400GB worth of storage that had been gobbled up. I came across this and a combination of find /var/lib/docker/containers/ -type f -name “*.log” -delete and docker-compose down && docker-compose up -d saw the reclamation of more than 390GB as shown below.
send 2
However, you guessed it, I ended up with a fresh install of central🤦‍♂️. I tried restoring a backup from 10 days ago following instructions from here but I keep getting this error. NB The backup file is almost 12GB

Any ideas how I could at least force the backup restore?

Hi Anthony,

There is a bug in Central where tmp files are not deleted. It will be fixed in the next release, but see Errors with sending forms to ODK Central because hard drive is full for a workaround that we shared a month ago.

In your attempt to resolve the tmp issue, you ran docker-compose down which is unfortunate because it will reset your database. We warn about the dangers of running this command at https://docs.getodk.org/central-install and provide recovery instructions at https://docs.getodk.org/central-troubleshooting/#troubleshooting-docker-compose-down. Try the recovery steps and report back with what you find.

As to the backup restore, that does seem like a bug and I'll ask the team to look into it. Before we force anything, have you successfully restored backups of less than 10GB before?

Yaw

1 Like

Hello @yanokwa,
I'll be sure to check out the recovery instructions in the next minute or so. Will revert with anything I find.

On the matter of backup restoration, this is my first time trying to restore a backup. So no, I have never even tried to restore a backup in the past.

:partying_face::partying_face::partying_face::partying_face:
Everything is back!
Thanks a ton!

One question though. Is there any overhead associated with the long list of instances displayed when you run ```docker volumes ls``?

1 Like

Hi Anthony,

Have you encountered any issues after restore? Im having issues with my forms after backup restore, all forms cannot be open via preview nor new submission. But submitted data are intact, have plenty of free space, root partition also have enough space.

Br

The component that powers the preview and web form submissions stores some information outside the core database that we backup. If that data is also not restored, you will run into some of the issues you are describing. I recently ran into this issue and we'll be providing some guidance on this limitation in our backups in the upcoming Central v1.2 release.

If you have full machine backups, it might be good to try that restore to see if that works better. If you don't and you have access to a recent copy of the ~/central folder, I can provide some high-level guidance on how to get previews working again.

Hi Yaw,

Yes I have a copy/backup of the central folder and no full machine backups. It would be very much appreciated if you can give guidance on how to make new submissions and form previews work again.

Br

How exactly did you restore? If you have the existing central folder and you are restoring into the same folder as described at https://docs.getodk.org/central-backup/#restoring-a-backup, the previews should work. Is that what you did?

Hi Yaw, yes as described at https://docs.getodk.org/central-backup/#restoring-a-backup, copied the central folder to the new central server and executed the intstructions. Forms and data submissions are intact but preview and new submissions still cant be accessed.

One way to fix the broken previews and web submissions is to download the XLSForm of each form, change the version number, and re-upload the form. This will generate entirely new previews and web submissions links and is a very quick fix if you don't have an ongoing campaign.

If you have distributed public access links and have an ongoing campaign that uses those links, then new previews and web submissions links might be too disruptive. In that case, you'll need to dump the Enketo databases and secrets from the old machine and restore it to the new machine so you can have your old previews and web submission links.

The instructions below assume that you are not changing the externally facing hostname (e.g., mycentralserver.example.com). If you are, you'll need find and replace the hostnames in the Enketo databases. Doing that is possible, but is out of scope for this post.

I strongly recommend you have a snapshot of both machines before taking any of the following steps!

On the old machine, make sure Central is running as usual. Run the following commands to dump the databases and secrets.

docker exec enketo_redis_main redis-cli -p 6379 save;
docker cp enketo_redis_main:/data/enketo-main.rdb ~/enketo-main.rdb;
docker exec enketo_redis_cache redis-cli -p 6380 save;
docker cp enketo_redis_cache:/data/enketo-cache.rdb ~/enketo-cache.rdb;
docker cp service:/etc/secrets ~/;

Now transfer the enketo-main.rdb, enketo-cache.rdb, secrets files from the home folder of the old machine to the home folder of the new machine. Make sure Central is running as usual, then run the following.

docker stop enketo_redis_main;
docker cp ~/enketo-main.rdb enketo_redis_main:/data/enketo-main.rdb;
docker start enketo_redis_main;
docker stop enketo_redis_cache;
docker cp ~/enketo-cache.rdb enketo_redis_cache:/data/enketo-cache.rdb;
docker start enketo_redis_cache;
docker stop service;
docker cp ~/secrets service:/etc;
docker start service;

In your secrets folder, you'll have three files: enketo-api-key, enketo-less-secret, enketo-secret. You'll need to use nano to edit your Enketo config to those values.

docker exec -it enketo sh;
apt-get install nano -y;
nano config/config.json;

Update config/config.json with the secrets values.

{
    "app name": "Enketo",
    "base path": "-",
    "encryption key": "contents of enketo-secret",
    "id length": 31,
    "less secure encryption key": "contents of enketo-less-secret",
    "linked form and data server": {
        "api key": "contents of enketo-api-key",

Now uninstall nano, and exit enketo container.

apt-get remove nano -y;
exit;

Now restart enketo.

docker restart enketo;
1 Like