# Central Backup Restore

**URL:** <https://forum.getodk.org/t/central-backup-restore/33047>\
**Category:** Support\
**Created:** [April 1, 2021, 3:08pm UTC](https://forum.getodk.org/t/central-backup-restore/33047 "2021-04-01T15:08:25Z")\
**Posts on this page:** 10\
**Page:** 1

<div class="post-metadata">

**Author:** ![aankrah](https://getodk.b-cdn.net/user_avatar/forum.getodk.org/aankrah/32/12857_2.png) [@aankrah](https://forum.getodk.org/u/aankrah)\
**Post date:** [April 1, 2021, 3:08pm UTC](https://forum.getodk.org/t/central-backup-restore/33047/1 "2021-04-01T15:08:25Z")

</div>

To start things off, I am running **Central client (v1.0.0), server (v1.0.3)** on **Ubuntu 18.04.5 LTS**

App users reported an error preventing them from submitting forms to central.  
 ![error](https://getodk.b-cdn.net/uploads/default/original/2X/d/d963ca3b0749b2fed38943defeb86893a71d1728.png)  
On the server, running `df -h` revealed the root partition was full.

 ![disk full](https://getodk.b-cdn.net/uploads/default/original/2X/9/95b80cb6801782f23e053f1c6a6085953e054ce6.png)  
Further investigation revealed that the `/var/lib/docker/overlay2/` folder had taken upwards of **420GB** as can be seen in the image above.

The traditional `docker system prune` and `docker system prune -a` could only salvage 210MB of storage which enabled app users to submit their data to the server. I however needed to find a way to release the over **400GB** worth of storage that had been gobbled up. I came across [this](https://forums.docker.com/t/some-way-to-clean-up-identify-contents-of-var-lib-docker-overlay/30604/33) and a combination of `find /var/lib/docker/containers/ -type f -name “*.log” -delete` and `docker-compose down && docker-compose up -d` saw the reclamation of more than **390GB** as shown below.  
 ![send 2](https://getodk.b-cdn.net/uploads/default/original/2X/1/1222e4bd0ccb4e673ae74a6d683e54e631e152fc.jpeg)  
However, you guessed it, I ended up with a fresh install of central🤦‍♂️. I tried restoring a backup from 10 days ago following instructions from [here](https://docs.getodk.org/central-backup/) but I keep getting this error. **NB** The backup file is almost **12GB**

 ![restore fail](https://getodk.b-cdn.net/uploads/default/original/2X/c/caec58cd5b493c80bb235dc0b01254d21c648790.png)

Any ideas how I could at least force the backup restore?

---

<div class="post-metadata">

**Author:** ![yanokwa](https://getodk.b-cdn.net/user_avatar/forum.getodk.org/yanokwa/32/1695_2.png) [@yanokwa](https://forum.getodk.org/u/yanokwa)\
**Post date:** [April 1, 2021, 6:07pm UTC](https://forum.getodk.org/t/central-backup-restore/33047/2 "2021-04-01T18:07:04Z")

</div>

Hi Anthony,

There is a bug in Central where tmp files are not deleted. It will be fixed in the next release, but see [Errors with sending forms to ODK Central because hard drive is full](https://forum.getodk.org/t/errors-with-sending-forms-to-odk-central-because-hard-drive-is-full/32661/5) for a workaround that we shared a month ago.

In your attempt to resolve the tmp issue, you ran `docker-compose down` which is unfortunate because it will reset your database. We warn about the dangers of running this command at [https://docs.getodk.org/central-install](https://docs.getodk.org/central-install) and provide recovery instructions at [https://docs.getodk.org/central-troubleshooting/#troubleshooting-docker-compose-down](https://docs.getodk.org/central-troubleshooting/#troubleshooting-docker-compose-down). Try the recovery steps and report back with what you find.

As to the backup restore, that does seem like a bug and I'll ask the team to look into it. Before we force anything, have you successfully restored backups of less than 10GB before?

Yaw

---

<div class="post-metadata">

**Author:** ![aankrah](https://getodk.b-cdn.net/user_avatar/forum.getodk.org/aankrah/32/12857_2.png) [@aankrah](https://forum.getodk.org/u/aankrah)\
**Post date:** [April 1, 2021, 6:14pm UTC](https://forum.getodk.org/t/central-backup-restore/33047/3 "2021-04-01T18:14:59Z")

</div>

Hello @yanokwa,  
I'll be sure to check out the recovery instructions in the next minute or so. Will revert with anything I find.

On the matter of backup restoration, this is my first time trying to restore a backup. So no, I have never even tried to restore a backup in the past.

---

<div class="post-metadata">

**Author:** ![aankrah](https://getodk.b-cdn.net/user_avatar/forum.getodk.org/aankrah/32/12857_2.png) [@aankrah](https://forum.getodk.org/u/aankrah)\
**Post date:** [April 1, 2021, 6:40pm UTC](https://forum.getodk.org/t/central-backup-restore/33047/4 "2021-04-01T18:40:35Z")

</div>

> [@yanokwa](#):
>
> [https://docs.getodk.org/central-troubleshooting/#troubleshooting-docker-compose-down](https://docs.getodk.org/central-troubleshooting/#troubleshooting-docker-compose-down). Try the recovery steps and report back with what you find.

🥳🥳🥳🥳  
Everything is back!  
Thanks a ton!

One question though. Is there any overhead associated with the long list of instances displayed when you run ```docker volumes ls``?

---

<div class="post-metadata">

**Author:** ![julio.wong](https://getodk.b-cdn.net/letter_avatar_proxy/v4/letter/j/96bed5/32.png) [@julio.wong](https://forum.getodk.org/u/julio.wong)\
**Post date:** [May 6, 2021, 8:32pm UTC](https://forum.getodk.org/t/central-backup-restore/33047/5 "2021-05-06T20:32:03Z")

</div>

Hi Anthony,

Have you encountered any issues after restore? Im having issues with my forms after backup restore, all forms cannot be open via preview nor new submission. But submitted data are intact, have plenty of free space, root partition also have enough space.

Br

---

<div class="post-metadata">

**Author:** ![yanokwa](https://getodk.b-cdn.net/user_avatar/forum.getodk.org/yanokwa/32/1695_2.png) [@yanokwa](https://forum.getodk.org/u/yanokwa)\
**Post date:** [May 6, 2021, 9:00pm UTC](https://forum.getodk.org/t/central-backup-restore/33047/6 "2021-05-06T21:00:52Z")

</div>

The component that powers the preview and web form submissions stores some information outside the core database that we backup. If that data is also not restored, you will run into some of the issues you are describing. I recently ran into this issue and we'll be providing some guidance on this limitation in our backups in the upcoming Central v1.2 release.

If you have full machine backups, it might be good to try that restore to see if that works better. If you don't and you have access to a recent copy of the ~/central folder, I can provide some high-level guidance on how to get previews working again.

---

<div class="post-metadata">

**Author:** ![julio.wong](https://getodk.b-cdn.net/letter_avatar_proxy/v4/letter/j/96bed5/32.png) [@julio.wong](https://forum.getodk.org/u/julio.wong)\
**Post date:** [May 7, 2021, 2:53am UTC](https://forum.getodk.org/t/central-backup-restore/33047/7 "2021-05-07T02:53:20Z")

</div>

Hi Yaw,

Yes I have a copy/backup of the central folder and no full machine backups. It would be very much appreciated if you can give guidance on how to make new submissions and form previews work again.

Br

---

<div class="post-metadata">

**Author:** ![yanokwa](https://getodk.b-cdn.net/user_avatar/forum.getodk.org/yanokwa/32/1695_2.png) [@yanokwa](https://forum.getodk.org/u/yanokwa)\
**Post date:** [May 7, 2021, 10:50pm UTC](https://forum.getodk.org/t/central-backup-restore/33047/8 "2021-05-07T22:50:20Z")

</div>

How exactly did you restore? If you have the existing `central` folder and you are restoring into the same folder as described at [https://docs.getodk.org/central-backup/#restoring-a-backup](https://docs.getodk.org/central-backup/#restoring-a-backup), the previews should work. Is that what you did?

---

<div class="post-metadata">

**Author:** ![julio.wong](https://getodk.b-cdn.net/letter_avatar_proxy/v4/letter/j/96bed5/32.png) [@julio.wong](https://forum.getodk.org/u/julio.wong)\
**Post date:** [May 8, 2021, 3:12am UTC](https://forum.getodk.org/t/central-backup-restore/33047/9 "2021-05-08T03:12:34Z")

</div>

Hi Yaw, yes as described at [https://docs.getodk.org/central-backup/#restoring-a-backup](https://docs.getodk.org/central-backup/#restoring-a-backup), copied the central folder to the new central server and executed the intstructions. Forms and data submissions are intact but preview and new submissions still cant be accessed.

---

<div class="post-metadata">

**Author:** ![yanokwa](https://getodk.b-cdn.net/user_avatar/forum.getodk.org/yanokwa/32/1695_2.png) [@yanokwa](https://forum.getodk.org/u/yanokwa)\
**Post date:** [May 8, 2021, 9:07pm UTC](https://forum.getodk.org/t/central-backup-restore/33047/10 "2021-05-08T21:07:35Z")

</div>

One way to fix the broken previews and web submissions is to download the XLSForm of each form, change the version number, and re-upload the form. This will generate entirely new previews and web submissions links and is a very quick fix if you don't have an ongoing campaign.

If you have distributed public access links and have an ongoing campaign that uses those links, then new previews and web submissions links might be too disruptive. In that case, you'll need to dump the Enketo databases and secrets from the old machine and restore it to the new machine so you can have your old previews and web submission links.

The instructions below assume that you are not changing the externally facing hostname (e.g., [mycentralserver.example.com](http://mycentralserver.example.com)). If you are, you'll need find and replace the hostnames in the Enketo databases. Doing that is possible, but is out of scope for this post.

**I strongly recommend you have a snapshot of both machines before taking any of the following steps!**

On the old machine, make sure Central is running as usual. Run the following commands to dump the databases and secrets.

```auto
docker exec enketo_redis_main redis-cli -p 6379 save;
docker cp enketo_redis_main:/data/enketo-main.rdb ~/enketo-main.rdb;
docker exec enketo_redis_cache redis-cli -p 6380 save;
docker cp enketo_redis_cache:/data/enketo-cache.rdb ~/enketo-cache.rdb;
docker cp service:/etc/secrets ~/;

```

Now transfer the `enketo-main.rdb`, `enketo-cache.rdb`, `secrets` files from the home folder of the old machine to the home folder of the new machine. Make sure Central is running as usual, then run the following.

```auto
docker stop enketo_redis_main;
docker cp ~/enketo-main.rdb enketo_redis_main:/data/enketo-main.rdb;
docker start enketo_redis_main;
docker stop enketo_redis_cache;
docker cp ~/enketo-cache.rdb enketo_redis_cache:/data/enketo-cache.rdb;
docker start enketo_redis_cache;
docker stop service;
docker cp ~/secrets service:/etc;
docker start service;

```

In your `secrets` folder, you'll have three files: `enketo-api-key`, `enketo-less-secret`, `enketo-secret`. You'll need to use nano to edit your Enketo config to those values.

```auto
docker exec -it enketo sh;
apt-get install nano -y;
nano config/config.json;

```

Update `config/config.json` with the `secrets` values.

```json
{
    "app name": "Enketo",
    "base path": "-",
    "encryption key": "contents of enketo-secret",
    "id length": 31,
    "less secure encryption key": "contents of enketo-less-secret",
    "linked form and data server": {
        "api key": "contents of enketo-api-key",

```

Now uninstall nano, and exit enketo container.

```auto
apt-get remove nano -y;
exit;

```

Now restart enketo.

```auto
docker restart enketo;

```