ODK Central Unexpectedly Goes Offline / Inaccessible

Tyler_Depke · October 2, 2026, 2:37pm

1. What is the issue? Please be detailed.
I'm self hosting ODK Central on a DigitalOcean Droplet and in the last week or so, ODK Central has gone offline twice. When I say "has gone offline" I mean I cannot access ODK Central to login via a web browser with an error "This site can’t be reached" and ODK Collect doesn't seem to have any successful communication with Central.

2. What steps can we take to reproduce this issue?
Not sure...

3. What have you tried to fix the issue?
When this happened a few days ago, I did a hard reset on my droplet and after a few minutes ODK Central was back up, no problems. Now, a few days later, ODK Central has gone offline again. Note, I have NEVER had this issue before and there haven't been any active submissions/admin activity/etc. in the last several days until noticing this issue.

In ODK Collect if I try to submit a survey, sending fails, pressing the refresh icon in the "Start New Form" Screen has a ! inside the refresh button and a "Form Update Failed" Notification pushes.

I've logged into DigitalOcean to verify the machine is running as I can see Activity in CPU %, Memory %, Bandwidth, etc. I can ping the server IP from my machine successfully AND successfully ping the exact website URL I'd put in a web browser. These two things lead me to believe this is an ODK Central Issue, not a Digital Ocean issue.

From Digital Oceans Web Console I opened the linux command line and ran:

docker compose stop && docker compose up -d and while that completed successfully, that didn't fix my problem.

htop and there were a couple alternating processes that were showing activity seen in the screen shots below:

I'd like to find the underlying problem here ASAP instead of just power cycling to put a bandaid on what might be a bigger issue.

4. Upload any forms or screenshots you can share publicly below.

jniles · October 2, 2026, 4:47pm

Hey @Tyler_Depke ,

I'm sorry you are having these issues! Could you add a few more details:

What ODK Central version are you running? How long since your last update?
How much free space is left on your machine (df -h)?
Where is your domain hosted - on Digital Ocean or on an external provider (e.g. Cloudflare)?
1. You might try using a combination of traceroute and curl to make sure you are actually reaching your server.

To further narrow down if it is ODK or a machine issue, you could see if you can access something else over the web on your server that is not ODK. The quickest way I can think of is using a one-line python script to serve a directory on your machine, such as python -m http.server --directory YOUR_DIRECTORY_NAME.

Best of luck!

Tyler_Depke · October 2, 2026, 6:53pm

Domain is via NameCheap
Ubuntu 25.04 x64
ODK Central version v2026.2.0-1-g3e9b78f via
cd /path/to/odk/central
git describe --tags --always

Using df -h shows me this:

root@Ubuntu-24LTS:~/central# df -h
Filesystem      Size  Used Avail Use% Mounted on
tmpfs           197M  1.8M  196M   1% /run
/dev/vda1        48G   28G   21G  58% /
tmpfs           984M     0  984M   0% /dev/shm
tmpfs           5.0M     0  5.0M   0% /run/lock
tmpfs           1.0M     0  1.0M   0% /run/credentials/systemd-journald.service
tmpfs           984M     0  984M   0% /tmp
tmpfs           1.0M     0  1.0M   0% /run/credentials/systemd-resolved.service
/dev/vda13      989M  128M  794M  14% /boot
/dev/vda15      105M  6.1M   99M   6% /boot/efi
tmpfs           1.0M     0  1.0M   0% /run/credentials/systemd-networkd.service
tmpfs           1.0M     0  1.0M   0% /run/credentials/serial-getty@ttyS0.service
tmpfs           1.0M     0  1.0M   0% /run/credentials/getty@tty1.service
tmpfs           197M   12K  197M   1% /run/user/0

After about 20min with AI this is what I arrived at:

So the problem is not ODK Central, Sentry, or DigitalOcean DNS. The Snap-packaged Docker daemon is being prevented by AppArmor from sending the DNS packets that containers need.

That also explains everything we've observed:

Host → DNS                  works
Host → Sentry HTTPS         works
Host → Docker Hub HTTPS     works

Container → DNS             blocked by AppArmor
        ↓
Docker embedded DNS         cannot resolve names
        ↓
NGINX cannot resolve Sentry
        ↓
NGINX exits
        ↓
central-nginx-1 restarts forever
        ↓
port 443 unavailable
I would NOT disable AppArmor globally

And I would not start randomly changing firewall rules now. We have very specific evidence identifying the blocker.

The clean long-term solution is likely to move Docker from the Snap package to Docker's normal Debian/Ubuntu package. But because this is your production ODK Central installation, I would not uninstall Snap Docker yet.

The one thing AI had me change was
nano /var/snap/docker/3613/config/daemon.json

{
    "log-level": "error",
    "dns": [
        "xx.xxx.xx.2",
        "xx.xxx.xx.3"
    ]
}

replacing the x's with real IP addresses, but it looks like NGINX is still in a restart loop.

AI referenced this article but not sure how relevant it is:

github.com/canonical/docker-snap

Can't pull image from registry on Ubuntu 26.04

opened 08:00AM - 13 Feb 26 UTC

closed 02:18PM - 22 Jun 26 UTC

jpm-canonical

Bug upstream

### Bug description Using the latest/stable version of this snap on Ubuntu 26.0…4, there is a permission denied error when trying to run the hello-world container. ``` $ docker run hello-world Unable to find image 'hello-world:latest' locally docker: Error response from daemon: Get "https://registry-1.docker.io/v2/": dial tcp: lookup registry-1.docker.io on 127.0.0.53:53: write udp 127.0.0.1:51264->127.0.0.53:53: write: permission denied Run 'docker run --help' for more information ``` ### To reproduce * Install Ubuntu 26.04 and fully update it. * `sudo snap install docker` * Add user to docker group and reboot * Try to run the hello-world container ### Environment information for bug reproduction ```shell Snap information: Name Version Rev Tracking Publisher Notes docker 28.4.0 3377 latest/stable canonical✓ - snapd 2.73 25935 latest/stable canonical✓ snapd Plug Connections: Interface Plug Slot Notes content - docker:config-ro - content - docker:docker-executables - content - docker:docker-registry-certificates - content[gpu-2404] docker:gpu-2404 mesa-2404:gpu-2404 - content docker:graphics-core22 - - docker docker:docker-cli docker:docker-daemon - docker-support docker:privileged :docker-support - docker-support docker:support :docker-support - firewall-control docker:firewall-control :firewall-control - home docker:home :home - log-observe docker:log-observe - - network docker:network :network - network-bind docker:network-bind :network-bind - network-control docker:network-control :network-control - opengl docker:opengl :opengl - removable-media docker:removable-media - - AppArmor version: AppArmor parser version 5.0.0~alpha1 Copyright (C) 1999-2008 Novell Inc. Copyright 2009-2018 Canonical Ltd. OS release: PRETTY_NAME="Ubuntu Resolute Raccoon (development branch)" NAME="Ubuntu" VERSION_ID="26.04" VERSION="26.04 (Resolute Raccoon)" VERSION_CODENAME=resolute ID=ubuntu ID_LIKE=debian HOME_URL="https://www.ubuntu.com/" SUPPORT_URL="https://help.ubuntu.com/" BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/" PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy" UBUNTU_CODENAME=resolute LOGO=ubuntu-logo Kernel and architecture: 6.19.0-3-generic #3-Ubuntu SMP PREEMPT_DYNAMIC Fri Jan 23 20:01:24 UTC 2026 x86_64 ``` ### Additional information ``` Feb 13 09:56:55 systemd[4329]: Started snap.docker.docker-00a12f15-cbca-4390-8632-7c89b705134a.scope. Feb 13 09:56:55 kernel: audit: type=1400 audit(1770969415.592:520): apparmor="DENIED" operation="open" class="file" profile="snap.docker.docker" name="/proc/7209/mountinfo" pid=7209 comm="snap-exec" requested_mask="r" denied_mask="r" fsuid=1000 ouid=1000 Feb 13 09:56:55 kernel: audit: type=1400 audit(1770969415.613:521): apparmor="DENIED" operation="file_perm" class="net" profile="snap.docker.dockerd" pid=3233 comm="dockerd" laddr=127.0.0.1 lport=48714 faddr=127.0.0.53 fport=53 family="inet" sock_type="dgram" protocol=17 requested="send" denied="send" Feb 13 09:56:55 kernel: audit: type=1400 audit(1770969415.613:522): apparmor="DENIED" operation="file_perm" class="net" profile="snap.docker.dockerd" pid=3233 comm="dockerd" laddr=127.0.0.1 lport=36161 faddr=127.0.0.53 fport=53 family="inet" sock_type="dgram" protocol=17 requested="send" denied="send" Feb 13 09:56:55 kernel: audit: type=1400 audit(1770969415.613:523): apparmor="DENIED" operation="file_perm" class="net" profile="snap.docker.dockerd" pid=3233 comm="dockerd" laddr=127.0.0.1 lport=48714 faddr=127.0.0.53 fport=53 family="inet" sock_type="dgram" protocol=17 requested="send" denied="send" Feb 13 09:56:55 kernel: audit: type=1400 audit(1770969415.613:524): apparmor="DENIED" operation="file_perm" class="net" profile="snap.docker.dockerd" pid=3233 comm="dockerd" laddr=127.0.0.1 lport=36161 faddr=127.0.0.53 fport=53 family="inet" sock_type="dgram" protocol=17 requested="send" denied="send" Feb 13 09:56:55 kernel: audit: type=1400 audit(1770969415.613:525): apparmor="DENIED" operation="file_perm" class="net" profile="snap.docker.dockerd" pid=3233 comm="dockerd" laddr=127.0.0.1 lport=38151 faddr=127.0.0.53 fport=53 family="inet" sock_type="dgram" protocol=17 requested="send" denied="send" Feb 13 09:56:55 kernel: audit: type=1400 audit(1770969415.613:526): apparmor="DENIED" operation="file_perm" class="net" profile="snap.docker.dockerd" pid=3233 comm="dockerd" laddr=127.0.0.1 lport=38151 faddr=127.0.0.53 fport=53 family="inet" sock_type="dgram" protocol=17 requested="send" denied="send" Feb 13 09:56:55 kernel: audit: type=1400 audit(1770969415.613:527): apparmor="DENIED" operation="file_perm" class="net" profile="snap.docker.dockerd" pid=3233 comm="dockerd" laddr=127.0.0.1 lport=51264 faddr=127.0.0.53 fport=53 family="inet" sock_type="dgram" protocol=17 requested="send" denied="send" Feb 13 09:56:55 kernel: audit: type=1400 audit(1770969415.614:528): apparmor="DENIED" operation="file_perm" class="net" profile="snap.docker.dockerd" pid=3233 comm="dockerd" laddr=127.0.0.1 lport=51264 faddr=127.0.0.53 fport=53 family="inet" sock_type="dgram" protocol=17 requested="send" denied="send" Feb 13 09:56:55 docker.dockerd[3233]: time="2026-02-13T09:56:55.616449497+02:00" level=error msg="unexpected HTTP error handling" error="<nil>" Feb 13 09:56:55 docker.dockerd[3233]: time="2026-02-13T09:56:55.616474707+02:00" level=error msg="Handler for POST /v1.51/images/create returned error: Get \"https://registry-1.docker.io/v2/\": dial tcp: lookup registry-1.docker.io on 127.0.0.53:53: write udp 127.0.0.1:51264->127.0.0.53:53: write: permission denied" ``` ``` = AppArmor = Time: 2026-02-13T11:5 Log: apparmor="DENIED" operation="open" class="file" profile="snap.docker.docker" name="/proc/124172/mountinfo" pid=124172 comm="snap-exec" requested_mask="r" denied_mask="r" fsuid=1000 ouid=1000 File: /proc/124172/mountinfo (read) Suggestions: * adjust program to not access '@{PROC}/@{pid}/mountinfo' * add one of 'mount-control, mount-observe, steam-support' to 'plugs' = AppArmor = Time: 2026-02-13T11:5 Log: apparmor="DENIED" operation="open" class="file" profile="snap.docker.docker" name="/proc/124172/mountinfo" pid=124172 comm="docker" requested_mask="r" denied_mask="r" fsuid=1000 ouid=1000 File: /proc/124172/mountinfo (read) Suggestions: * adjust program to not access '@{PROC}/@{pid}/mountinfo' * add one of 'mount-control, mount-observe, steam-support' to 'plugs' = AppArmor = Time: 2026-02-13T11:5 Log: apparmor="DENIED" operation="file_perm" class="net" profile="snap.docker.dockerd" pid=122423 comm="dockerd" laddr=127.0.0.1 lport=57283 faddr=127.0.0.53 fport=53 family="inet" sock_type="dgram" protocol=17 requested="send" denied="send" = AppArmor = Time: 2026-02-13T11:5 Log: apparmor="DENIED" operation="file_perm" class="net" profile="snap.docker.dockerd" pid=122423 comm="dockerd" laddr=127.0.0.1 lport=36273 faddr=127.0.0.53 fport=53 family="inet" sock_type="dgram" protocol=17 requested="send" denied="send" = AppArmor = Time: 2026-02-13T11:5 Log: apparmor="DENIED" operation="file_perm" class="net" profile="snap.docker.dockerd" pid=122423 comm="dockerd" laddr=127.0.0.1 lport=57283 faddr=127.0.0.53 fport=53 family="inet" sock_type="dgram" protocol=17 requested="send" denied="send" = AppArmor = Time: 2026-02-13T11:5 Log: apparmor="DENIED" operation="file_perm" class="net" profile="snap.docker.dockerd" pid=122423 comm="dockerd" laddr=127.0.0.1 lport=36273 faddr=127.0.0.53 fport=53 family="inet" sock_type="dgram" protocol=17 requested="send" denied="send" = AppArmor = Time: 2026-02-13T11:5 Log: apparmor="DENIED" operation="file_perm" class="net" profile="snap.docker.dockerd" pid=122423 comm="dockerd" laddr=127.0.0.1 lport=55082 faddr=127.0.0.53 fport=53 family="inet" sock_type="dgram" protocol=17 requested="send" denied="send" = AppArmor = Time: 2026-02-13T11:5 Log: apparmor="DENIED" operation="file_perm" class="net" profile="snap.docker.dockerd" pid=122423 comm="dockerd" laddr=127.0.0.1 lport=38912 faddr=127.0.0.53 fport=53 family="inet" sock_type="dgram" protocol=17 requested="send" denied="send" ```

Has anyone else had issues with docker? I have version 29.8.0. What concerns me is this issue occurred without me doing any sort of updates, at least to my knowledge. The first instance of the server being down occurred Wed, Sep 23, 2026.

Tyler_Depke · October 2, 2026, 7:15pm

Here is a screenshot from Digital Ocean looking at the activity over the last 7 days.

This clear abrupt change makes me think there was some sort of update scheduled for 00:00 on Sept 30 may have kicked off this change in activity.

yanokwa · October 2, 2026, 9:29pm

What do your container logs say?

Tyler_Depke · October 3, 2026, 11:11am

Looks like a looping with NGINX caused by an issue with how Docker was installed (Docker Snap) and AI kept suggesting an issue with AppArmor. I reverted docker back to an older version, but that still didn't solve the issue.

Since it was taking so long to find a solution and I found out that my machine was running Ubuntu 25 when I thought it was running Ubuntu 24 LTS, I just ended up creating a new droplet using Ubuntu 26 LTS and migrating my ODK Central install to the new droplet and now ODK Central runs fine without any issues.

That's probably unhelpful for anyone else who might run into this issue, but I've found sometimes the best solution isn't continuing to dig deeper when you hit rock, but find a new place to dig.

Full Log Here

root@Ubuntu-24LTS:~/central# docker logs --tail 100 central-service-1
at /usr/odk/lib/task/task.js:62:48
at /usr/odk/node_modules/@sentry/opentelemetry/build/cjs/resource-CTgtEOmw.js:604:15
at Module.handleCallbackErrors (/usr/odk/node_modules/@sentry/core/build/cjs/utils/handleCallbackErrors.js:11:26)
at /usr/odk/node_modules/@sentry/opentelemetry/build/cjs/resource-CTgtEOmw.js:603:19
at AsyncLocalStorage.run (node:internal/async_local_storage/async_context_frame:63:14)
at SentryAsyncLocalStorageContextManager.with (/usr/odk/node_modules/@sentry/opentelemetry/build/cjs/index.js:27:36)
at ContextAPI.with (/usr/odk/node_modules/@opentelemetry/api/build/src/api/context.js:51:46)
at Tracer.startActiveSpan (/usr/odk/node_modules/@opentelemetry/sdk-trace-base/build/src/Tracer.js:138:32)
at /usr/odk/node_modules/@sentry/opentelemetry/build/cjs/resource-CTgtEOmw.js:601:19
at /usr/odk/node_modules/@sentry/opentelemetry/build/cjs/resource-CTgtEOmw.js:740:21
at _startSpan (/usr/odk/node_modules/@sentry/opentelemetry/build/cjs/resource-CTgtEOmw.js:574:10)
at Object.startSpan (/usr/odk/node_modules/@sentry/opentelemetry/build/cjs/resource-CTgtEOmw.js:616:10)
at Module.startSpan (/usr/odk/node_modules/@sentry/core/build/cjs/tracing/trace.js:32:16)
at Command. (/usr/odk/lib/task/task.js:58:19)
generating local service configuration..
waiting for PostgreSQL to become connectable to...
running migrations..
logging server upgrade..
null
starting cron..
using 4 worker(s) based on available memory (2062970880)..
starting server.
2026-10-02T05:07:23: PM2 log: Launching in no daemon mode
2026-10-02T05:07:23: PM2 error: (node:57) [DEP0176] DeprecationWarning: fs.R_OK is deprecated, use fs.constants.R_OK instead
(Use node --trace-deprecation ... to show where the warning was created)
2026-10-02T05:07:23: PM2 log: App [service:0] starting in -cluster mode-
2026-10-02T05:07:23: PM2 log: App [service:0] online
2026-10-02T05:07:23: PM2 log: App [service:1] starting in -cluster mode-
2026-10-02T05:07:23: PM2 log: App [service:1] online
2026-10-02T05:07:23: PM2 log: App [service:2] starting in -cluster mode-
2026-10-02T05:07:24: PM2 log: App [service:2] online
2026-10-02T05:07:24: PM2 log: App [service:3] starting in -cluster mode-
2026-10-02T05:07:25: PM2 log: App [service:3] online
npm error path /usr/odk
npm error command failed
npm error signal SIGTERM
npm error command sh -c "pm2-runtime" ./pm2.config.js
npm error A complete log of this run can be found in: /root/.npm/_logs/2026-10-02T05_07_18_939Z-debug-0.log
generating local service configuration..
waiting for PostgreSQL to become connectable to...
running migrations..
logging server upgrade..
null
starting cron..
using 4 worker(s) based on available memory (2062970880)..
starting server.
2026-10-02T14:08:14: PM2 log: Launching in no daemon mode
2026-10-02T14:08:14: PM2 error: (node:57) [DEP0176] DeprecationWarning: fs.R_OK is deprecated, use fs.constants.R_OK instead
(Use node --trace-deprecation ... to show where the warning was created)
2026-10-02T14:08:14: PM2 log: App [service:0] starting in -cluster mode-
2026-10-02T14:08:14: PM2 log: App [service:0] online
2026-10-02T14:08:15: PM2 log: App [service:1] starting in -cluster mode-
2026-10-02T14:08:15: PM2 log: App [service:1] online
2026-10-02T14:08:15: PM2 log: App [service:2] starting in -cluster mode-
2026-10-02T14:08:15: PM2 log: App [service:2] online
2026-10-02T14:08:15: PM2 log: App [service:3] starting in -cluster mode-
2026-10-02T14:08:16: PM2 log: App [service:3] online
npm error path /usr/odk
npm error command failed
npm error signal SIGTERM
npm error command sh -c "pm2-runtime" ./pm2.config.js
npm error A complete log of this run can be found in: /root/.npm/_logs/2026-10-02T14_08_11_376Z-debug-0.log
generating local service configuration..
waiting for PostgreSQL to become connectable to...
running migrations..
logging server upgrade..
null
starting cron..
using 4 worker(s) based on available memory (2062970880)..
starting server.
2026-10-02T14:32:22: PM2 log: Launching in no daemon mode
2026-10-02T14:32:22: PM2 error: (node:57) [DEP0176] DeprecationWarning: fs.R_OK is deprecated, use fs.constants.R_OK instead
(Use node --trace-deprecation ... to show where the warning was created)
2026-10-02T14:32:22: PM2 log: App [service:0] starting in -cluster mode-
2026-10-02T14:32:23: PM2 log: App [service:0] online
2026-10-02T14:32:23: PM2 log: App [service:1] starting in -cluster mode-
2026-10-02T14:32:23: PM2 log: App [service:1] online
2026-10-02T14:32:23: PM2 log: App [service:2] starting in -cluster mode-
2026-10-02T14:32:23: PM2 log: App [service:2] online
2026-10-02T14:32:24: PM2 log: App [service:3] starting in -cluster mode-
2026-10-02T14:32:24: PM2 log: App [service:3] online
generating local service configuration..
waiting for PostgreSQL to become connectable to...
running migrations..
logging server upgrade..
null
starting cron..
using 4 worker(s) based on available memory (2062970880)..
starting server.
2026-10-02T18:21:56: PM2 log: Launching in no daemon mode
2026-10-02T18:21:56: PM2 error: (node:57) [DEP0176] DeprecationWarning: fs.R_OK is deprecated, use fs.constants.R_OK instead
(Use node --trace-deprecation ... to show where the warning was created)
2026-10-02T18:21:56: PM2 log: App [service:0] starting in -cluster mode-
2026-10-02T18:21:56: PM2 log: App [service:0] online
2026-10-02T18:21:56: PM2 log: App [service:1] starting in -cluster mode-
2026-10-02T18:21:57: PM2 log: App [service:1] online
2026-10-02T18:21:57: PM2 log: App [service:2] starting in -cluster mode-
2026-10-02T18:21:58: PM2 log: App [service:2] online
2026-10-02T18:21:58: PM2 log: App [service:3] starting in -cluster mode-
2026-10-02T18:22:00: PM2 log: App [service:3] online
root@Ubuntu-24LTS:~/central#

More AI troubleshooting led to:

Summary

This is now very clear. We have confirmed the root cause.

The Docker daemon is running under the AppArmor profile:

profile="snap.docker.dockerd"

and AppArmor is actively denying its DNS packets:

laddr=127.0.0.1
faddr=127.0.0.53
fport=53
requested="send"
denied="send"

So the problem is not ODK Central, not NGINX, not Sentry, and not the Central version.

It is:

Docker Snap → AppArmor → DNS

And importantly, this is happening even after reverting from Docker 29.8.0 to 29.6.1, so the rollback cannot fix the underlying issue.

LN · October 8, 2026, 11:35pm

Words to live by. Who knew we'd get such wisdom on the ODK forum!!