Errors with sending forms to ODK Central because hard drive is full

[Jneethli0504]

1. What is the problem? Be very detailed.
I am trying to send forms to ODK Central from a device, but whenever I do, I get the following error:

Error: Generic Exception: Error: Internal Server Error (500) at (server address)

If I look in the Backup log on the server, I see this:
{"stack":"Error: ENOSPC: no space left on device, mkdir '/tmp/tmp-147AbmdANZfLi4J'","message":"ENOSPC: no space left on device, mkdir '/tmp/tmp-147AbmdANZfLi4J'","success":false}

2. What app or server are you using and on what device and operating system? Include version numbers.
I am using ODK Collect v1.30.1 on a Samsung Galaxy Grand Prime Pro
My server is hosted by Digital Ocean

3. What you have you tried to fix the problem?
I did try sending one form at a time, and that worked for 3 of the forms. However, it's stopped working now. I did not really want to try anything else for fear of breaking the server.

I was wondering if I have no space on my ODK Central server, so I wanted to delete some of the form entries, many of which have photos attached. However, I don't see a way on ODK Central to delete submissions that I have already downloaded.

[jpringle]

It looks like your server has run out of disk space. Can you verify by logging in to the terminal / console and running this command df -h. See more here: https://www.digitalocean.com/community/questions/how-can-i-tell-how-much-disk-space-is-available-on-my-droplet

If it is your droplet that has run out of space, do you have budget to increase the disk space for your droplet? See this article: https://www.digitalocean.com/docs/droplets/how-to/resize/

If you are the $5 per month droplet and resize to the $10 per month droplet (if using US dollars), then the droplet would increase its storage from 25GB to 50GB.

The ODK Docs have a page about adding a volume on Digital Ocean to get extra storage: https://docs.getodk.org/central-install-digital-ocean/#adding-external-storage

What do you think about these options before going in and deleting survey data?

[yanokwa]

We’ve seen servers get in a state where their /tmp directory is not cleared. It’s something we have been troubleshooting and are hoping to have a fix for soon. It disproportionately affects servers that receive media and those that run Google Drive backups.

Clearing the /tmp file is relatively quick than adding more storage but there are a few different possible paths to take depending on just how full the drive is. If you’re comfortable with an ODK core team member logging into your server, that will be the quickest way to get a fix. You can email support@getodk.org (or send me a personal message on the forum) if you’d like to go in that direction.

[LN]

I believe clearing /tmp from the service container should always work.

First, make sure you have a backup of your data. This should be a safe process but there’s always some risk when doing direct filesystem manipulations. If you don’t use managed backups, you can configure a temporary Google Drive account, request an immediate backup and then disable backing up again. You can also use the API to request a backup. See https://docs.getodk.org/central-backup/#performing-an-immediate-backup

Then log into your server and go to where you have your Central folder. Then get a shell to your service container:

cd
cd central
docker exec -it service sh

From there, go to the tmp directory and remove the subdirectories in it. They should all have a tmp prefix. Be very sure you are in the correct directory because rm cannot be undone.

cd /tmp
rm -rf tmp*

You should then be able to exit the container’s shell and restart Central:

exit
docker-compose stop
docker-compose up -d

It's important to run the last two docker-compose commands because it appears the temporary file library we use maintains a reference to submission temporary directories. Without the restart, submissions will fail.

[yanokwa]

We've found a safer way to remove these stubborn temp files. The commands are:

cd ~/central;
docker exec service sh -c "find /tmp/tmp-* -type f -mmin +60 -delete";

The 60 ensures you only delete files that are older than 60 minutes. This does not require a restart because it doesn't delete the temporary folders, only their contents.

[Billie]

Hi, I'm in the same position but worse, I don't have enough space to run docker, any tips so I can clear something to get it to start and remove the tmp files?

[yanokwa]

Try the standard Ubuntu things...

• sudo apt-get clean; sudo apt-get autoclean; sudo apt-get autoremove;
• Temporarily disable swap
• Remove things in /tmp

[Florian_May]

There's a command line tool called ncdu which lets you find large files.
Tutorial e.g. https://lintut.com/ncdu-check-disk-usage/

[Billie]

Thank you, we are back up and running!