Disk full issue with central-service container

[lv]

Hi everyone,

1. What is the issue? Please be detailed.

I was having issues with ODK Central: I couldn’t log in or access the data. When I checked the server, I found that the disk was full.

After inspecting, I saw that the container central-service-1 was taking up a lot of space in Docker’s overlay storage (/var/lib/docker/overlay2/********/diff).

3. What have you tried to fix the issue?

I deleted the container and rebuilt it, but I don’t really understand why this happened or how to prevent central-service-1 from filling the disk again.

Does anyone know what might be causing this or how to avoid it in the future?

Thanks for any help!

[lv]

I found that the disk usage is coming from inside the central-service container. Inspecting the overlay filesystem shows that almost all the growth is happening in /tmp, which has grown to ~15 GB:

8.0K   ./data
8.0K   ./etc
8.0K   ./run
24K    ./usr
200K   ./var
324K   ./root
15G    ./tmp

It seems /tmp inside the container can grow without bounds over time.
Is it safe/recommended to manage this by mounting /tmp as a volume or by setting a size limit (e.g. using tmpfs) in docker-compose?

Thanks!

[yanokwa]

There was a bug in Central about five years ago where tmp wasn't being cleared. What version of Central are you running?

[lv]

versions:
(v2025.3.1)
client (v2025.3.1)
server (v2025.3.0)

Is this the last one?

Thanks

[yanokwa]

v2025 is pretty new so it's unlikely to be the tmp bug.

Before changing the docker setup, can you look through tmp to see exactly what is in there? I'd use ncdu because it gives you a user-friendly way to explore usage.

# login to service container
$ docker exec -it central-service-1 bash

# install and run ncdu
$ apt-get update; apt-get install ncdu
$ ncdu /tmp

# remove ncdu and exit container
$ apt-get remove -y ncdu
$ exit

[lv]

Greetings!

Large temporary folders, about 15 GB each, appear every day. I'm deleting the old ones daily. And they look like this:

​​The problem is that it doesn't appear as recyclable; the container just keeps getting bigger.

​The problem is that it doesn't appear as recyclable; the container just keeps getting bigger.

Thanks for the help!

[yanokwa]

dat.gz files suggest that you are making backups with the Central API. Are you? Are those backups completing successfully?

[lv]

Yes, I always back up the system and save it to Google Drive... Sometimes it's been a lifesaver in emergencies. But I also create disk instances. Is it no longer usable? I haven't restored that backup in a long time; I just keep it.