Central disk is suddenly full, unable to correct

1. What is the problem? Be very detailed.
My 60GB disk at Digital Ocean suddenly became 100% continuously full during a scheduled daily backup to Google Docs. Surveys can no longer be uploaded to Central. Because of the full disk, I could not perform Digital Ocean's required console update and log onto the console.

2. What app or server are you using and on what device and operating system? Include version numbers.
ODK Central v1.2 on a Digital Ocean droplet.

3. What you have you tried to fix the problem?
Following guidance from Digital Ocean support, I rebooted the droplet to the Recovery ISO, mounted my files and used chroot to access my file system.

The "cd central" command returned a directory not found error error. "pwd" revealed that the current directory was /. With FileZilla, I found the Central directory at /mnt/root/central.

After a cd to /mnt/root/central, the "docker-compose --version" command confirmed my previous docker upgrade, returning "docker-compose version 1.29.2, build 5becea4c."

I then attempted to upgrade from Central version from v1.2 to v1.3. Previously, running upgrade cleared tmp files and freed disk space (even if the version was current). However, the "git pull" command returned "fatal: Unable to read current working directory: No such file or directory." I reconfirmed with pdw that the working directory was still /mnt/root/central.

Next, I attempted to follow guidance from the ODK Forum page "Errors with sending forms to ODK Central because hard drive is full". The command "docker exec -it service sh" returned the following error: "Cannot connect to the Docker daemon at unix://var/run/docker.sock. Is the docker daemon running?" This is puzzling, because docker worked fine for other commands.

Any help or guidance to solve this disk full problem would be greatly appreciated.

4. What steps can we take to reproduce the problem?
The attached screenshot shows a sharp peak in disk use when the daily backup to Google Docs takes place. Although the backup is compressed, little actual reduction takes place because almost all the data size is the jpg media file in each survey instance. It appears that the problem occurs when the backup attempts to use more than 100% of the droplet disk space. Although in this case the backup failed because Google Docs was full, a failed backup a few months ago, when fewer survey instances had been uploaded, did not cause the problem with a continuously full Digital Ocean disk..

5. Anything else we should know or have? If you have a test form or screenshots or logs, attach below.

UPDATE: The problem is not exactly solved, but when I rebooted from Recovery ISO back to Hard Drive, the troublesome tmp files have mysteriously disappeared, the disk has plenty of free space, and survey instances can be submitted. To prevent a recurrence of the disk full problem, I have disabled Central's backup to Google Drive and substituted another backup method.

1 Like

Glad the disk has space now, @Hayden_Boyd. I'd strongly recommend setting up monitoring as described at https://docs.getodk.org/central-install-digital-ocean/#setting-up-monitoring.

I'd also consider using SnapShooter to setup more frequent snapshots than Digital Ocean allows.

If you do want to continue using Central's backups, I'd keep a minimum of 2.5x the size of your database available as free space.

1 Like

Thanks, Yaw. As suggested, I've set up a monitoring alert for disk utilization. For backup, I'm now using Windows Task Scheduler to run a Python script that invokes the Briefcase CLI to download daily all new survey instances to my local drive, from whence they are backed up automatically to an external hard drive and online to iDrive. This is working well, and it does not depend on me remembering to frequently clear out Google Drive before it fills up.