It has happened to me twice, so I thought to consult the forum. I deployed a central server with 250GB disk space. The forms were supposed to collect images as well. I kept an eye on the data size, which was around 30~40GB for some time. But then one day the server started misbehaving, and I discovered that the disk is full somehow (full 250GB). Since I had the form submission stats from some days before, and not much was added in between, I knew that something else happened. I expanded the disk space (server is on AWS) to set it to 500GB, and meanwhile also upgraded the central server to v1.5.3. After the whole upgrade process which I think also involves some raw data cleaning, the data size got back to 50GB, which is the real size.
I thought perhaps its a one-off thing, but yesterday again I came across another central installation choking the 8GB disk, which literally had no data submissions on the server. I had to expand the disk here as well; however I could not figure out how to optimize the data size, or why it is happening.
Can someone help with this? Are there some docker logs doing this? Or is it happening inside postgresql tables, etc? Is there some command which could clean up the disk to get rid of extra data? I am also worried about leaving the server unmanaged for couple of months, because of the fear of disk choking.
Let me know if I could provide more traces. I have the 8GB server still intact with bloated data.
What version of Central is running on the 8 GB server? If it's running a version below Central v1.2, then you could have this issue of a growing disk. As always, we recommend running the latest version of Central.
In general, 8 GB is not enough space for anything and it doesn't give you much buffer if things go wrong. 20 GB is a much more reasonable minimum disk space target. I'll include a note in our docs about recommended minimum disk space.
To figure out if Docker is the cause of this, run docker system df. A good way to explore disk space usage on the machine itself is running ncdu /. You'll likely need to apt-get install ncdu first.
It's best practice to have monitoring of CPU, RAM, disk, and processes for uptime monitoring, regular security updates, and automated backups. If you have that in place, then I suppose you don't have to do much to the server. If you don't, then yes, you can get into all sorts of trouble.
If you have verified working full machine backups, you can also try methods to reduce space usage:
If Docker containers and images are taking up lots of space, you can consider running docker system prune after successful upgrades to remove dangling images and stopped containers. The prune command will remove stopped containers on your machine (not just related to Central), so be careful about running it!
If you've manually deleted a lot of records in the database (which we don't recommend), a DB vacuum might be in order. A full vacuum requires 2x the disk space and locks the table, think carefully before attempting.
I have not tried other commands yet. Let me know if some diagnosis could be made before I change more things (I am queued to upgrade to v1.5.4 as well).
If you run docker system prune, you should get 1.4 GB back on your machine. The rest of the disk space usage you report is the data you are storing in Central's database.
To give you a sense of the minimum usage, this is what I expect on a new install with an external database.
root@example:~/central# docker system df
TYPE TOTAL ACTIVE SIZE RECLAIMABLE
Images 6 6 3.546GB 0B (0%)
Containers 7 7 1.289MB 0B (0%)
Local Volumes 9 8 30.85kB 0B (0%)
Build Cache 0 0 0B 0B