Enketo not loading

Support
1

1. What is the issue? Please be detailed.
I upgraded ODK central to latest version 2023.4 in multiple steps from 1.2.2
All forms and data are showing and I am able to downlaod submissions
However the ENKETO editing is not working
I get a circular pulsing bubble in middle and then I get an error

2. What steps can we take to reproduce this issue?
I am behind a reverse proxy where SSL termination happens

My configurations are shared here

.env

DOMAIN=survey.xxx.yyyu

# Used for Let's Encrypt expiration emails and Enketo technical support emails
SYSADMIN_EMAIL=XX@YYY.COM

# Options: letsencrypt, customssl, upstream, selfsign
SSL_TYPE=upstream

# Do not change if using SSL_TYPE=letsencrypt
HTTP_PORT=80
HTTPS_PORT=443
UPSTREAM_HTTPS_PORT=443

docker-compose.yml

  service:
...
    environment:
      - HTTPS_PORT=${UPSTREAM_HTTPS_PORT:-443}
.....
  nginx:
    build:
      context: .
      args:
        - OIDC_ENABLED=${OIDC_ENABLED:-false}
      dockerfile: nginx.dockerfile
    depends_on:
      - service
      - enketo
    environment:
      - DOMAIN=${DOMAIN}
      - CERTBOT_EMAIL=${SYSADMIN_EMAIL}
      - SSL_TYPE=${SSL_TYPE:-letsencrypt}
      - SENTRY_ORG_SUBDOMAIN=${SENTRY_ORG_SUBDOMAIN:-o130137}
    ports:
      - "${HTTP_PORT:-80}:80"
      - "${HTTPS_PORT:-443}:443"
....

  enketo:
    environment:
      - DOMAIN=${DOMAIN}
      - HTTPS_PORT=${UPSTREAM_HTTPS_PORT:-443}

A URL call is failing as per the Developer tools

POST   https://survey.xxx.yyyu/-/transform/xform/XXXXXXXXXXXXXXXXXXXXXXXXXXX

Status  504
VersionHTTP/2
Transferred338 B (160 B size)
Referrer Policysame-origin
DNS ResolutionSystem

The reverse proxy is getting a "gateway Timed Out" from the enketo/ Collect server

I have tried various combinations of ports in .env to no avail

2

Further, I have logged to Reverse proxy Server when trying to edit a specific submission ID. I am getting following timeouts
.env

HTTP_PORT=80
HTTPS_PORT=443

CLIENT_USER_IP_ADD - "GET /v1/projects/7/forms/FORM___NAME/submissions/uuid%3A80ebee26-XXXXXXXXXXXXXXXXXXXXXXXXa1b31fad8a42/edit HTTP/2.0" 302 696 "https://survey.xxx.yyyu/" 

CLIENT_USER_IP_ADD - - [05/Dec/2023:16:11:05 +0530] "GET /-/edit/HmXXXXXXXXXXXXXXXXXXq6T6?instance_id=uuid:80ebee26-d29e-XXXXXXXXXXXXXXXXXXXXXXXa1b31fad8a42&return_url=https%3A%2F%2Fsurvey.xxx.yyyu%2F%23%2Fprojects%2F7%2Fforms%2FFORM___NAME%2Fsubmissions%2Fuuid%3A80ebee26-XXXXXXXXXXXXXXXXXXXXXXXXa1b31fad8a42 HTTP/2.0" 304 0 "https://survey.xxx.yyyu/" 

CLIENT_USER_IP_ADD -  "GET /-/fonts/OpenSans-Regular-webfont.woff HTTP/2.0" 304 0 "https://survey.xxx.yyyu/-/edit/HmXXXXXXXXXXXXXXXXXXq6T6?instance_id=uuid:80ebee26-d29e-XXXXXXXXXXXXXXXXXXXXXXXa1b31fad8a42&return_url=https%3A%2F%2Fsurvey.xxx.yyyu%2F%23%2Fprojects%2F7%2Fforms%2FFORM___NAME%2Fsubmissions%2Fuuid%3A80ebee26-XXXXXXXXXXXXXXXXXXXXXXXXa1b31fad8a42" 

CLIENT_USER_IP_ADD - - "GET /-/fonts/fontawesome-webfont.woff?v=4.6.2 HTTP/2.0" 304 0 "https://survey.xxx.yyyu/-/edit/HmXXXXXXXXXXXXXXXXXXq6T6?instance_id=uuid:80ebee26-d29e-XXXXXXXXXXXXXXXXXXXXXXXa1b31fad8a42&return_url=https%3A%2F%2Fsurvey.xxx.yyyu%2F%23%2Fprojects%2F7%2Fforms%2FFORM___NAME%2Fsubmissions%2Fuuid%3A80ebee26-XXXXXXXXXXXXXXXXXXXXXXXXa1b31fad8a42" 

CLIENT_USER_IP_ADD -  "GET /-/css/theme-kobo.css HTTP/2.0" 304 0 "https://survey.xxx.yyyu/-/edit/HmXXXXXXXXXXXXXXXXXXq6T6?instance_id=uuid:80ebee26-d29e-XXXXXXXXXXXXXXXXXXXXXXXa1b31fad8a42&return_url=https%3A%2F%2Fsurvey.xxx.yyyu%2F%23%2Fprojects%2F7%2Fforms%2FFORM___NAME%2Fsubmissions%2Fuuid%3A80ebee26-XXXXXXXXXXXXXXXXXXXXXXXXa1b31fad8a42" 

CLIENT_USER_IP_ADD -  "GET /-/js/build/enketo-webform-edit.js HTTP/2.0" 304 0 "https://survey.xxx.yyyu/-/edit/HmXXXXXXXXXXXXXXXXXXq6T6?instance_id=uuid:80ebee26-d29e-XXXXXXXXXXXXXXXXXXXXXXXa1b31fad8a42&return_url=https%3A%2F%2Fsurvey.xxx.yyyu%2F%23%2Fprojects%2F7%2Fforms%2FFORM___NAME%2Fsubmissions%2Fuuid%3A80ebee26-XXXXXXXXXXXXXXXXXXXXXXXXa1b31fad8a42" 

CLIENT_USER_IP_ADD - -"GET /-/fonts/OpenSans-Bold-webfont.woff HTTP/2.0" 304 0 "https://survey.xxx.yyyu/-/edit/HmXXXXXXXXXXXXXXXXXXq6T6?instance_id=uuid:80ebee26-d29e-XXXXXXXXXXXXXXXXXXXXXXXa1b31fad8a42&return_url=https%3A%2F%2Fsurvey.xxx.yyyu%2F%23%2Fprojects%2F7%2Fforms%2FFORM___NAME%2Fsubmissions%2Fuuid%3A80ebee26-XXXXXXXXXXXXXXXXXXXXXXXXa1b31fad8a42" 

CLIENT_USER_IP_ADD - "GET /-/css/theme-kobo.print.css HTTP/2.0" 200 2382 "https://survey.xxx.yyyu/-/edit/HmXXXXXXXXXXXXXXXXXXq6T6?instance_id=uuid:80ebee26-d29e-XXXXXXXXXXXXXXXXXXXXXXXa1b31fad8a42&return_url=https%3A%2F%2Fsurvey.xxx.yyyu%2F%23%2Fprojects%2F7%2Fforms%2FFORM___NAME%2Fsubmissions%2Fuuid%3A80ebee26-XXXXXXXXXXXXXXXXXXXXXXXXa1b31fad8a42" 

CLIENT_USER_IP_ADD - "GET /-/locales/build/en/translation-combined.json HTTP/2.0" 304 0 "https://survey.xxx.yyyu/-/edit/HmXXXXXXXXXXXXXXXXXXq6T6?instance_id=uuid:80ebee26-d29e-XXXXXXXXXXXXXXXXXXXXXXXa1b31fad8a42&return_url=https%3A%2F%2Fsurvey.xxx.yyyu%2F%23%2Fprojects%2F7%2Fforms%2FFORM___NAME%2Fsubmissions%2Fuuid%3A80ebee26-XXXXXXXXXXXXXXXXXXXXXXXXa1b31fad8a42" 

CLIENT_USER_IP_ADD - "GET /-/submission/HmXXXXXXXXXXXXXXXXXXq6T6?enketoId=HmXXXXXXXXXXXXXXXXXXq6T6&instanceId=uuid%3A80ebee26-XXXXXXXXXXXXXXXXXXXXXXXXa1b31fad8a42 HTTP/2.0" 304 0 "https://survey.xxx.yyyu/-/edit/HmXXXXXXXXXXXXXXXXXXq6T6?instance_id=uuid:80ebee26-d29e-XXXXXXXXXXXXXXXXXXXXXXXa1b31fad8a42&return_url=https%3A%2F%2Fsurvey.xxx.yyyu%2F%23%2Fprojects%2F7%2Fforms%2FFORM___NAME%2Fsubmissions%2Fuuid%3A80ebee26-XXXXXXXXXXXXXXXXXXXXXXXXa1b31fad8a42" 

CLIENT_USER_IP_ADD - "POST /-/transform/xform/HmXXXXXXXXXXXXXXXXXXq6T6 HTTP/2.0" 504 160 "https://survey.xxx.yyyu/-/edit/HmXXXXXXXXXXXXXXXXXXq6T6?instance_id=uuid:80ebee26-d29e-XXXXXXXXXXXXXXXXXXXXXXXa1b31fad8a42&return_url=https%3A%2F%2Fsurvey.xxx.yyyu%2F%23%2Fprojects%2F7%2Fforms%2FFORM___NAME%2Fsubmissions%2Fuuid%3A80ebee26-XXXXXXXXXXXXXXXXXXXXXXXXa1b31fad8a42"

So the primary error is the 504 Gateway Timeout that is happening

3

Looked into logs on nginx

docker compose logs nginx  --tail 50 -f | grep a66ef3669d8e

I am seeing following 504 error

central-nginx-1  | My.Reverse.Proxy.P- - [05/Dec/2023:11:22:29 +0000] "POST /-/transform/xform/XXXXXXXXXXXXXXXXXXXXXXXX HTTP/1.0" 504 160 "https://survey.xxx.yyyu/-/edit/Ulp7SPsyJzrUGN8aUXmBjiJzeB60xTp?instance_id=uuid:XXXXXXXXXXx-XXXX-XXXX-XXXX-a66ef3669d8e&return_url=https%3A%2F%2Fsurvey.xxx.yyyu%2F%23%2Fprojects%2F7%2Fforms%2FORMS%2Fsubmissions%2Fuuid%XXXXXXXXXXx-XXXX-XXXX-XXXX-a66ef3669d8e" "Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:120.0) Gecko/20100101 Firefox/120.0" "My.Client.IP.Address"

I have also disabled the UFW on local VM and that did not help

4

Dear community members,
Has anyone faced this issue or similar? How were you able to resolve it?

6

I have managed to RESOLVE The Error using various suggestions in the forum

Step 1: Check reach ability of ODK host to the ODK central website https://xxx.yyyy.zzzz

cd ~/central 
curl  -i https://xxx.yyyy.zzzz 
curl  https://xxx.yyyy.zzzz  | grep getodk.org

Potential issues:
The server is host is not able to reach itself due to
a. DNS issues:

nslookup xxx.yyyy.zzzz

You should get the public IP address of the ODK central website xxx.yyyy.zzzz
If Not, check reachability to DNS and configuration of DNS. It may happen that you are using a lcoal DNS server and not global DNS server and the lcoal DNS server does not have the correct DNS records

b. Firewall issues
Check that the server is able to reach the public IP of ODK central host

ping 11.22.33.44

After completion of step 1, you should be able to ensure that the ODK central host is able to reach the website https://xxx.yyyy.zzzz

curl https://xxx.yyyy.zzzz | grep getodk.org

Step 2: Check Reach-ability of ENKETO service to the ODK central website https://xxx.yyyy.zzzz

cd ~/central
docker exec -it central-enketo-1 bash
# You will enter the enketo container  and can execute commands in it
# root@707a18658765868:/srv/src/enketo/packages/enketo-express
# Curl is not normally installed in container
apt install curl dnsutils
nslookup xxx.yyyy.zzzz
curl  https://xxx.yyyy.zzzz | grep -c getodk.org

You should be able to get the public IP of the https://xxx.yyyy.zzzz using nslookup
Lets say the public IP is 11.22.33.44
If not, then the docker container is not able to resolve the DNS address and you may need to edit the /etc/docker/daemon.json file on the HOST (Not in the container). However I find it simpler to edit the /etc/hosts file on the host

# Exit from container to the host
exit
# Edit docker dameon
sudo nano /etc/docker/daemon.json

{
    "dns": ["1.2.3.4", "9.8.7.6", "8.8.8.8"]
}

docker compose stop
systemctl restart docker 
docker compose up -d


# OR, Edit the hosts file and add your public IP / Reverse Proxy IP and website  to local resolver
sudo nano  /etc/hosts

127.0.0.1 localhost
127.0.1.1 odk_svr
# REPLACE WITH YOUR PUBLIC IP AND ODK CENTRAL WEBSITE ADDRESS
11.22.33.44  xxx.yyyy.zzzz

# The following lines are desirable for IPv6 capable hosts
::1     ip6-localhost ip6-loopback
fe00::0 ip6-localnet
ff00::0 ip6-mcastprefix
ff02::1 ip6-allnodes
ff02::2 ip6-allrouters

# RESTART from STEP 2

In my case, i added my reverse proxy IP to the /etc/hosts on the ODK host

NOW AM A HAPPY CAMPER

Thanks for all those who contributed
Cheers
Vivek

7

My configurations are

NGINX Upstream Reverse Proxy**

server {
 listen 443 ssl http2;
 server_name xxx.yyy.zzzz;
 ssl_certificate /etc/nginx/ssl/bundle.crt;
 ssl_certificate_key /etc/nginx/ssl/bundle.key;
 ssl_prefer_server_ciphers on;
 ssl_protocols TLSv1 TLSv1.1 TLSv1.2;

 location / {
       proxy_pass "http://192.168.14.XX:8080";
       proxy_set_header Host $host;
       proxy_set_header X-Real-IP $remote_addr;
       proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
       proxy_set_header X-Forwarded-Proto $scheme;
       proxy_buffering off;

        client_max_body_size 500M;
        proxy_connect_timeout       300;
        proxy_send_timeout          300;
        proxy_read_timeout          120;
        send_timeout                300;
      }
 }

server {
 listen 80;
 server_name xxx.yyy.zzzz;
 return 301 https://$host$request_uri;
}

ODK Central Host

.env

DOMAIN=xxx.yyy.zzzz
SYSADMIN_EMAIL=xxx.yyy.zzzz@gmail.xxx.yyy.zzzz
SSL_TYPE=upstream
HTTP_PORT=8080
HTTPS_PORT=8443
UPSTREAM_HTTPS_PORT=443

docker-compose.yml

Only change is in lines as recommended in docs
HTTPS_PORT=${UPSTREAM_HTTPS_PORT:-443}

services:
  postgres14:
    build:
      context: .
      dockerfile: postgres14.dockerfile
    volumes:
      - postgres14:/var/lib/odk/postgresql/14
    environment:
      POSTGRES_USER: odk
      POSTGRES_PASSWORD: odk
      POSTGRES_DB: odk
    restart: always
  postgres:
    # This service upgrades from postgres 9.6 to 14.
    # The legacy name must be maintained to allow access to the anonymous volume.
    build:
      context: .
      dockerfile: postgres-upgrade.dockerfile
    volumes:
      - /var/lib/postgresql/data
      - postgres14:/var/lib/postgresql/14
      - ./files/postgres14/upgrade:/postgres14-upgrade
    environment:
      PGUSER: odk
      POSTGRES_INITDB_ARGS: -U odk
      POSTGRES_PASSWORD: odk
      POSTGRES_DB: odk
  mail:
    image: "ixdotai/smtp:v0.5.2"
    volumes:
      - ./files/mail/rsa.private:/etc/exim4/dkim.key.temp:ro
    environment:
      - MAILNAME=${DOMAIN}
      - DKIM_KEY_PATH=/etc/exim4/dkim.key.temp
    restart: always
  service:
    build:
      context: .
      dockerfile: service.dockerfile
    depends_on:
      - secrets
      - postgres14
      - mail
      - pyxform
      - enketo
    volumes:
      - secrets:/etc/secrets
      - /data/transfer:/data/transfer
    environment:
      - DOMAIN=${DOMAIN}
      - SYSADMIN_EMAIL=${SYSADMIN_EMAIL}
      - HTTPS_PORT=${UPSTREAM_HTTPS_PORT:-443}
      - NODE_OPTIONS=${SERVICE_NODE_OPTIONS:-}
      - DB_HOST=${DB_HOST:-postgres14}
      - DB_USER=${DB_USER:-odk}
      - DB_PASSWORD=${DB_PASSWORD:-odk}
      - DB_NAME=${DB_NAME:-odk}
      - DB_SSL=${DB_SSL:-null}
      - EMAIL_FROM=${EMAIL_FROM:-no-reply@$DOMAIN}
      - EMAIL_HOST=${EMAIL_HOST:-mail}
      - EMAIL_PORT=${EMAIL_PORT:-25}
      - EMAIL_SECURE=${EMAIL_SECURE:-false}
      - EMAIL_IGNORE_TLS=${EMAIL_IGNORE_TLS:-true}
      - EMAIL_USER=${EMAIL_USER:-}
      - EMAIL_PASSWORD=${EMAIL_PASSWORD:-}
      - OIDC_ENABLED=${OIDC_ENABLED:-false}
      - OIDC_ISSUER_URL=${OIDC_ISSUER_URL:-}
      - OIDC_CLIENT_ID=${OIDC_CLIENT_ID:-}
      - OIDC_CLIENT_SECRET=${OIDC_CLIENT_SECRET:-}
      - SENTRY_ORG_SUBDOMAIN=${SENTRY_ORG_SUBDOMAIN:-o130137}
      - SENTRY_KEY=${SENTRY_KEY:-3cf75f54983e473da6bd07daddf0d2ee}
      - SENTRY_PROJECT=${SENTRY_PROJECT:-1298632}
    command: [ "wait-for-it", "${DB_HOST:-postgres14}:5432", "--", "./start-odk.sh" ]
    restart: always
    logging:
      driver: local
  nginx:
    build:
      context: .
      args:
        - OIDC_ENABLED=${OIDC_ENABLED:-false}
      dockerfile: nginx.dockerfile
    depends_on:
      - service
      - enketo
    environment:
      - DOMAIN=${DOMAIN}
      - CERTBOT_EMAIL=${SYSADMIN_EMAIL}
      - SSL_TYPE=${SSL_TYPE:-letsencrypt}
      - SENTRY_ORG_SUBDOMAIN=${SENTRY_ORG_SUBDOMAIN:-o130137}
      - SENTRY_KEY=${SENTRY_KEY:-3cf75f54983e473da6bd07daddf0d2ee}
      - SENTRY_PROJECT=${SENTRY_PROJECT:-1298632}
    ports:
      - "${HTTP_PORT:-80}:80"
      - "${HTTPS_PORT:-443}:443"
    healthcheck:
      test: [ "CMD-SHELL", "nc -z localhost 80 || exit 1" ]
    restart: always
    logging:
      driver: local
      options:
        max-file: "30"
  pyxform:
    image: 'ghcr.io/getodk/pyxform-http:v2.0.3'
    restart: always
  secrets:
    volumes:
      - secrets:/etc/secrets
    build:
      context: .
      dockerfile: secrets.dockerfile
    command: './generate-secrets.sh'
  enketo:
    volumes:
      - secrets:/etc/secrets
    build:
      context: .
      dockerfile: enketo.dockerfile
    restart: always
    depends_on:
      - secrets
      - enketo_redis_main
      - enketo_redis_cache
    environment:
      - DOMAIN=${DOMAIN}
      - SUPPORT_EMAIL=${SYSADMIN_EMAIL}
      - HTTPS_PORT=${UPSTREAM_HTTPS_PORT:-443}
  enketo_redis_main:
    image: redis:7.2
    volumes:
      - ./files/enketo/redis-enketo-main.conf:/usr/local/etc/redis/redis.conf:ro
      - enketo_redis_main:/data
    command:
      - redis-server
      - /usr/local/etc/redis/redis.conf
    restart: always
  enketo_redis_cache:
    image: redis:7.2
    volumes:
      - ./files/enketo/redis-enketo-cache.conf:/usr/local/etc/redis/redis.conf:ro
      - enketo_redis_cache:/data
    command:
      - redis-server
      - /usr/local/etc/redis/redis.conf
    restart: always
volumes:
  secrets:
  transfer:
  postgres14:
  enketo_redis_main:
  enketo_redis_cache

DNS Configuration in Host

I added my Nginx reverse proxy IP to the local resolver for my website.
Say the reverse proxy IP is 11.22.33.4444 and ODK central wesbite is xxx.yyy.zzzz
This enabled website resolution on the ODK servers within the datacenter

sudo nano /etc/hosts

# Modify the following as per your environment
11.22.33.4444 xxx.yyy.zzzz

I hope these would help anyone who faces similar issues
Huge Thanks to all those who helped and pointed me in the right directions.

Dr Vivek Gupta

2 Likes