Upgrade to Central v2023.2 failed no space left on device

Support
1

I am upgrading a Central server from 1.5.2. The server runs on Digital Ocean with the default Database.

As the software is before V2023.2 I have been following the instructions to upgrade to this version https://docs.getodk.org/central-upgrade/#central-upgrade-2023-2

The process went smoothly until step 6 "building the code".

Note that under step 3, verifying the space available, I had to alter the check-available-space script as per others but to overcome this, I removed the exclamation mark (!) in this line
if ! docker compose version >/dev/null 2>/dev/null; then

And the disk space was verified

# ./files/postgres14/upgrade/check-available-space
[check-available-space]
[check-available-space]     Free space: 7.8GB
[check-available-space] Required space: 6.0GB
[check-available-space]
[check-available-space] You have enough space to upgrade.
[check-available-space]
[check-available-space] Continue with the instructions at https://docs.getodk.org/central-upgrade/
[check-available-space]

I followed the steps but it failed at
# docker compose build --pull
with insufficient space

> [nginx intermediate 5/5] RUN OIDC_ENABLED="false" files/prebuild/build-frontend.sh:
32.66 npm WARN deprecated stable@0.1.8: Modern JS already guarantees Array#sort() is a stable sort, so this library is deprecated. See the compatibility table on MDN: https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/Array/sort#browser_compatibility
42.82 npm WARN tar TAR_ENTRY_ERROR ENOSPC: no space left on device, write
42.84 npm ERR! code ENOSPC
42.84 npm ERR! syscall write
42.84 npm ERR! errno -28
42.84 npm ERR! nospc ENOSPC: no space left on device, write
42.84 npm ERR! nospc There appears to be insufficient space on your system to finish.
42.85 npm ERR! nospc Clear up some disk space and try again.
42.85
42.85 npm ERR! A complete log of this run can be found in: /root/.npm/_logs/2024-04-25T12_05_50_623Z-debug-0.log
------
failed to solve: process "/bin/sh -c OIDC_ENABLED=\"$OIDC_ENABLED\" files/prebuild/build-frontend.sh" did not complete successfully: exit code: 228

And now I don't have sufficient space:

# ./files/postgres14/upgrade/check-available-space
[check-available-space]
[check-available-space]     Free space: 412MB
[check-available-space] Required space: 6.0GB
[check-available-space]
[check-available-space] !!!
[check-available-space] !!! ERROR: You do not have enough space available to upgrade.
[check-available-space] !!!
[check-available-space] !!! Increase your disk by at least 5.6GB and try again.
[check-available-space] !!!
[check-available-space]

It is clear that I have not followed a step somewhere, but I don't know how I can recover this, expecially given that there must have bene a partial installation.

2

Take a snapshot of your current machine and try these commands on the snapshot to see if it frees up some space:

docker builder prune
docker image prune

If you have swap, you can also temporarily disable swap, delete the swapfile, and that should get you a bit more free space.

1 Like
3

Thanks @yanokwa those were good pointers, particularly working from a snapshot. The pruning did free up space, but not enough and whilst the docker build worked, it failed to spin up the postgres DB at docker compose up postgres

In the end I had to revert to the snapshot, get more disk space on DigitalOcean (which didn't actually cost more) and resumed the install at docker compose build --pull it worked fine with more disk space. Server is back up at the latest version and everything where I left it

If anybody else has this problem, inote that you may still encounter disk space issues downstream even if you pass the check-available-space script. But the problem is simple to rescue

2 Likes