# Backing up Postgres and Redis DBs

**URL:** <https://forum.getodk.org/t/backing-up-postgres-and-redis-dbs/43336>\
**Category:** Support\
**Tags:** odk-central\
**Created:** [October 6, 2023, 5:13am UTC](https://forum.getodk.org/t/backing-up-postgres-and-redis-dbs/43336 "2023-10-06T05:13:06Z")\
**Posts on this page:** 9\
**Page:** 1

<div class="post-metadata">

**Author:** ![Dr\_Vivek\_Gupta](https://getodk.b-cdn.net/user_avatar/forum.getodk.org/dr_vivek_gupta/32/15988_2.png) [@Dr\_Vivek\_Gupta](https://forum.getodk.org/u/Dr_Vivek_Gupta)\
**Post date:** [October 6, 2023, 5:13am UTC](https://forum.getodk.org/t/backing-up-postgres-and-redis-dbs/43336/1 "2023-10-06T05:13:06Z")

</div>

Dear friends,  
Many thanks to the ODK community. I have been a ODK Aggregate and now ODK central user for several years now. Currently we are collecting our Primary Eye Care and eye care survey related data using ODK central, which has around 150,300 records.  
So indeed what started has a hobbyist implementation by a public health practitioner doctor with interest in health IT has now firmly taken roots in our day-to-day operations !

The last ODK central installation was done 2 years back and hasn't been touched since. I am using  
ODK Central client v1.2.2  
ODK Central server v1.2.1  
The server is up and running on a local VMWare based VM in our institutes server stack on Ubuntu 20.04, behind common another VM running a reverse proxy. That reverse proxy serves many other subdomains as well, apart from our ODK instance.

1. 
  - I am looking to upgrade to current versions of Central.

2. 
  - I have taken a VM snapshot to be safe.

3. 
  - I have also downloaded all individual forms data and XLSform definitions.

In this regard, I wanted to take a server backup using API. I am using the Postman and have managed to

1. POST to the v1/session and get a Auth token
2. Get list of projects
3. Initiate a request to /v1/backup

However, When taking backup, I am getting an error:  
_"Could not get response_ _Error: Maximum response size reached"_

 ![Screenshot 2023-10-06 102206](https://getodk.b-cdn.net/uploads/default/original/3X/9/d/9d4e69a6b3e465989410ee50126136e078adcea2.png)

**Query:**

1. How do I download and save the backup file, now that i know how to make a HTTP call
2. I was able to download a backup from the web UI from [https://nationalttsurvey.aiims.edu/#/system/backups](https://nationalttsurvey.aiims.edu/#/system/backups) That was around 4 GB compressed. Is this the same as taking a API based backup ?
3. How do I backup the REDIS for Enketo

\*\*\*\*\*\* UPDATE  
Postman gives a SEND AND DOWNLOAD option that was helpful for individual small form data download. However Postman exited on me on a gigabyte sized download in between

 ![Screenshot 2023-10-06 133417](https://getodk.b-cdn.net/uploads/default/original/3X/2/5/25db633498c3ef9dda7966e604c561abbb840064.png)

Many Thanks for your support  
Dr Vivek Gupta, MD

---

<div class="post-metadata">

**Author:** ![Dr\_Vivek\_Gupta](https://getodk.b-cdn.net/user_avatar/forum.getodk.org/dr_vivek_gupta/32/15988_2.png) [@Dr\_Vivek\_Gupta](https://forum.getodk.org/u/Dr_Vivek_Gupta)\
**Post date:** [October 6, 2023, 6:08am UTC](https://forum.getodk.org/t/backing-up-postgres-and-redis-dbs/43336/2 "2023-10-06T06:08:34Z")

</div>

To answer my own question, I have made some headway using the following Python code

```python
from pprint import pprint
import requests
from requests.Exceptions import HTTPError

BASEURL = 'SERVERADDRESS/v1'
EMAIL = "myUserName"
PASSWORD = "myPassword"

# GET TOKEN
try:
    response = requests. Post(BASEURL + '/sessions', json={
            "email": EMAIL,
            "password": PASSWORD
    })
    response.raise_for_status()
    jsonResponse = response.json()
    TOKEN = jsonResponse["token"]
    # print("SAVED Token")
    print(TOKEN)
    headers = {
        'Authorization': 'Bearer ' + TOKEN,
    }

    # Get Number of projects
    try:
        response2 = requests.get(BASEURL + '/projects', headers=headers)
        print(f' Response Status Code: {response2.status_code}')
        jsonResponse2 = response2.json()
        pprint(jsonResponse2)

    except HTTPError as http_err:
        print(f'HTTP error occurred: {http_err}')
    except Exception as err:
        print(f'Other error occurred: {err}')

# Catch Auth Errors
except HTTPError as http_err:
    print(f'HTTP error occurred: {http_err}')
except Exception as err:
    print(f'Other error occurred: {err}')

```

Now off to trying to get backup and downloading it !

Vivek

---

<div class="post-metadata">

**Author:** ![Dr\_Vivek\_Gupta](https://getodk.b-cdn.net/user_avatar/forum.getodk.org/dr_vivek_gupta/32/15988_2.png) [@Dr\_Vivek\_Gupta](https://forum.getodk.org/u/Dr_Vivek_Gupta)\
**Post date:** [October 6, 2023, 7:16am UTC](https://forum.getodk.org/t/backing-up-postgres-and-redis-dbs/43336/3 "2023-10-06T07:16:38Z")

</div>

Following code to

1. Use token received from an AUTH request
2. Get list of all projects
3. Get list of forms in each project and their total submissions
4. Report total submissions per project and across projects

```auto
from pprint import pprint
import requests
from requests.exceptions import HTTPError

TOKEN = 'putyoursecrettokenhere'
BASEURL = 'serverURL/v1/'
headers = {
    'Authorization': 'Bearer ' + TOKEN,
}

xheaders = {
    'Authorization': 'Bearer ' + TOKEN,
    'X-Extended-Metadata': 'true',
}

total_submission = 0
try:
    r_projects = requests.get(BASEURL + 'projects', headers=headers)
    json_projects = r_projects.json()
    PROJECTS = len(json_projects)
    print(f'\nTOTAL PROJECTS: {PROJECTS}')
    for i in range(PROJECTS):
        # print(f'{json_projects[i]}')
        PROJECT = json_projects[i]
        project_id = str(PROJECT['id'])
        print(f"\nID = {project_id}")
        print(f"NAME = {PROJECT['name']}")
        print(f"Archived = {PROJECT['archived']}")
        project_submissions = 0
        try:
            r_forms = requests.get(BASEURL + 'projects/' + project_id + '/forms', headers=xheaders)
            json_forms = r_forms.json()
            FORMS = len(json_forms)

            print(f'TOTAL FORMS: {FORMS}')
            for j in range(FORMS):
                # print(f'{json_forms[j]}')
                form = json_forms[j]
                form_id = str(form['xmlFormId'])
                SUBMISSIONS = form['submissions']
                project_submissions = project_submissions + SUBMISSIONS
                total_submission = total_submission + SUBMISSIONS
                print(f" SUBMISSIONS: {form['submissions']} NAME: {form['name']} STATE: {form['state']} ")

        except HTTPError as http_err:
            print(f'HTTP error occurred: {http_err}')

        finally:
            print(f"PROJECT SUBMISSION: {project_submissions}")

except HTTPError as http_err:
    print(f'HTTP error occurred: {http_err}')
except Exception as err:
    print(f'Other error occurred: {err}')

finally:
    print(f"TOTAL SUBMISSIONS: {total_submission}")

```

TOTAL SUBMISSIONS: 151220

---

<div class="post-metadata">

**Author:** ![Florian\_May](https://getodk.b-cdn.net/user_avatar/forum.getodk.org/florian_may/32/7758_2.png) [@Florian\_May](https://forum.getodk.org/u/Florian_May)\
**Post date:** [October 6, 2023, 8:24am UTC](https://forum.getodk.org/t/backing-up-postgres-and-redis-dbs/43336/4 "2023-10-06T08:24:42Z")

</div>

Welcome to the ODK community!

A quick idea, apologies for brevity as I'm mobile.

Since you are using Python, would the pyodk package be useful for your purpose?  
Alternatively, you could use the R package ruODK to access amd download submissions.

When you run the backup, maybe increasing server size might address the error you're getting?

---

<div class="post-metadata">

**Author:** ![Dr\_Vivek\_Gupta](https://getodk.b-cdn.net/user_avatar/forum.getodk.org/dr_vivek_gupta/32/15988_2.png) [@Dr\_Vivek\_Gupta](https://forum.getodk.org/u/Dr_Vivek_Gupta)\
**Post date:** [October 6, 2023, 9:06am UTC](https://forum.getodk.org/t/backing-up-postgres-and-redis-dbs/43336/5 "2023-10-06T09:06:37Z")

</div>

Many Thanks  
I will definitely look at pyODK. The main concern is having a restorable backup.  
Meanwhile I am perusing writing a python script to download data and write to a file.

```auto
from pprint import pprint
import requests

TOKEN = 'TOKEN'
BASEURL = 'https://url.com/v1/backup'
headers = {
    'Authorization': 'Bearer ' + TOKEN,
}
response = requests.get(BASEURL, headers=headers, stream=True)
print(response.headers)

Content_Disposition = response.headers.get('Content-Disposition')
responseContentType = response.headers.get('Content-Type')

x = Content_Disposition.split("=")
responseFileName = str(x[1])
responseFileName = responseFileName.replace('"', '')
responseFileName = responseFileName.replace(':', '')

print(responseContentType)
print(responseFileName)

if responseContentType == 'application/zip':
    # 200 MB chunk size
    with open(responseFileName, mode="wb") as file:
        for chunk in response.iter_content(chunk_size=200 * 1024 * 1024):
            file.write(chunk)

```

So far, I have downloaded 1.5GB data and script is still working.  
I will get back with the status of downloaded file once completed

\*\*\* Update \*\*\*

Confirmed that the downloaded archive (5,10,21,41,782 bytes) was with errors (tested using 7zip)

Best Wishes  
Vivek

---

<div class="post-metadata">

**Author:** ![KagundaJM](https://getodk.b-cdn.net/user_avatar/forum.getodk.org/kagundajm/32/22887_2.png) [@KagundaJM](https://forum.getodk.org/u/KagundaJM)\
**Post date:** [October 7, 2023, 7:34am UTC](https://forum.getodk.org/t/backing-up-postgres-and-redis-dbs/43336/6 "2023-10-07T07:34:19Z")

</div>

Backup from the web UI is no longer available in latest version. As you have already taken a snapshot, I would recommend you upgrade to the latest Central version.

> > How do I backup the REDIS for Enketo

[You'll need to dump the Enketo databases and secrets from your server](https://forum.getodk.org/t/central-backup-restore/33047/10) by running the following commands:

```bash
  docker exec central-enketo_redis_main-1 redis-cli -p 6379 save;
  docker cp central-enketo_redis_main-1:/data/enketo-main.rdb ~/enketo-main.rdb;
  docker exec central-enketo_redis_cache-1 redis-cli -p 6380 save;
  docker cp central-enketo_redis_cache-1:/data/enketo-cache.rdb ~/enketo-cache.rdb;
  docker cp central-service-1:/etc/secrets ~/;

```

---

<div class="post-metadata">

**Author:** ![yanokwa](https://getodk.b-cdn.net/user_avatar/forum.getodk.org/yanokwa/32/1695_2.png) [@yanokwa](https://forum.getodk.org/u/yanokwa)\
**Post date:** [October 7, 2023, 12:19pm UTC](https://forum.getodk.org/t/backing-up-postgres-and-redis-dbs/43336/7 "2023-10-07T12:19:49Z")

</div>

@Dr_Vivek_Gupta, I would not use the API backup for this because it does not include the data in the Redis DB which is very helpful for a complete restoration. I would...

1. Restore the VM snapshot to another machine and see if you can upgrade that machine without any problems. If you can do this, you've confirmed you can have a smooth upgrade which is ultimately what you need to.

2. If you want to be extra safe, dump the Postgres DB and the Redis DBs (as @KagundaJM suggested) from the old machine and store it somewhere safe.

---

<div class="post-metadata">

**Author:** ![Dr\_Vivek\_Gupta](https://getodk.b-cdn.net/user_avatar/forum.getodk.org/dr_vivek_gupta/32/15988_2.png) [@Dr\_Vivek\_Gupta](https://forum.getodk.org/u/Dr_Vivek_Gupta)\
**Post date:** [October 7, 2023, 12:48pm UTC](https://forum.getodk.org/t/backing-up-postgres-and-redis-dbs/43336/8 "2023-10-07T12:48:25Z")

</div>

Many Thanks Kagunda and Yankowa for the concise instructions. These are quite self explanatory. May you also consider adding these to the main backup documentation page.  
I will be able to try them out on Monday as the server access is restricted.  
Best Wishes  
Vivek

---

<div class="post-metadata">

**Author:** ![Dr\_Vivek\_Gupta](https://getodk.b-cdn.net/user_avatar/forum.getodk.org/dr_vivek_gupta/32/15988_2.png) [@Dr\_Vivek\_Gupta](https://forum.getodk.org/u/Dr_Vivek_Gupta)\
**Post date:** [December 8, 2023, 5:49pm UTC](https://forum.getodk.org/t/backing-up-postgres-and-redis-dbs/43336/9 "2023-12-08T17:49:09Z")

</div>

Many thanks,  
With your directions, I have managed to complete upgrade to latest greatest version of Central. The support that you have provided was absolutely invaluable

Best wishes  
Vivek