Forms not downloading or rendering on reverse-proxied, tunneled ODK Central server

[jniles]

1. What is the issue? Please be detailed.
I have an ODK Central instance hosted on a machine behind a NAT. I've used Rathole to tunnel the ODK Central instance to a public IP (hosted on Digital Ocean) and everything works normally (DNS, form upload, login, project listing, email). However, when trying to preview forms or downloading on an Android device, I get "Could not connect with Form Server" and "Form download failed", respectively.

I have followed the debugging steps for DNS in https://docs.getodk.org/central-troubleshooting/#preview-could-not-connect-with-server to no success. I also can download the compiled XML, which indicates that the problem wasn't a silent fail in the form upgrade. When checking the ODK logs, the nginx server returns a "POST /-/transform/xform/[id string] HTTP/1.1" 500 error, and I cannot find more detailed logs than this.

Finally, because I'm using a tunnel through a reverse proxy, I have the "upstream" SSL configuration in the .env file, with both ports defined for HTTP and HTTPS.

I'm running ODK Central version 2023.4, the latest ODK Collect, and hosted on Debian 12 via docker compose.

2. What steps can we take to reproduce this issue?
You would need to set up a reverse proxy and tunnel to a local server that is running the docker-compose version of ODK Central, as I have described above.

3. What have you tried to fix the issue?
As mentioned above, I have tried to look into the logs, and I also followed the DNS debugging steps, thinking that it was a DNS issue.

4. Upload any forms or screenshots you can share publicly below.

I'm pretty savvy with Linux - how can I get better logs to see what might be causing the 500 failure? Is there an easy way to get those logs? I've looked through the enkento, service, and nginx containers to try and find a more detailed error message, but could not find any.

[ktuite]

Hi @jniles,

I'm not totally sure about your situation but I'll write some thoughts that I hope are useful clues and don't lead you astray...

1. POST /-/transform/xform/[id string] is about Central communicating with Enketo (- is a proxy to the enketo service running in its own container). I expect that 500 error to occur on form draft upload or publish. You might be able to look in that container to get more details about that particular 500 error, or enketo may not be running/available. If you can't preview forms through the Central website, that's another clue that something isn't set up right with enketo.

2. Not being able to download forms on Android is a separate problem and shouldn't be related to that 500 error. Collect uses the Central OpenRosa API over HTTPS to get the form lists/forms, so if it's not reaching the Central server, it could be an issue with ports and/or certificates. Knowing that, if you look up your errors in the rest of the ODK forum, you might find more clues, or at least other people also having a trouble running Central outside the normal (recommended) 1 machine + 1 public IP setup.

[jniles]

Thanks for the reply @ktuite! It gives me somewhere to start digging. I'm suspecting somewhere the PORTs are either being used in a way that conflicts with my port-forwarding schemes. I'll do some digging and maybe ping back when I have more details or questions.

Thanks!

[jniles]

Thanks @ktuite, this pointed me in the right direction. I now have the OpenRosa API correctly resolving forms to ODK Collect. I hadn't realized that it would inject the $PORT if the port was non-standard, but it makes sense that it would (e.g., the form list is resolves as $DOMAIN:$HTTPS_PORT/v1/...) from Collect.

If anyone is also running behind a reverse proxy, the way I solved this issue was to maintain the $HTTPS_PORT and $HTTP_PORT as 443 and 80 respectively, and manually update the docker-compose.yml to use my own non-standard ports (e.g. 8443, 8080). Collect looks for the forms at $DOMAIN, and my reverse proxy can handle the rest.

I'm still debugging the Enkento issue, but I don't have a use-case for Enkento at this time... it's just nice to have.

EDIT: Enkento is now fixed! I think it required a docker rebuild, and likely suffered from the same issue.

Thanks so much @ktuite!

[Dr_Vivek_Gupta]

Hi Niles
I am in a similar situation
Can you share the relevant portions of ,env and docker-compose for enketo please.
How did you rebuild enketo only? Did yo also have to rebuild redis ?
Thanks
Vivek

[nmambre]

@jniles

If anyone is also running behind a reverse proxy, the way I solved this issue was to maintain the $HTTPS_PORT and $HTTP_PORT as 443 and 80 respectively, and manually update the docker-compose.yml to use my own non-standard ports (e.g. 8443, 8080). Collect looks for the forms at $DOMAIN, and my reverse proxy can handle the rest.

Hi @jniles
Which portions of the docker compose did you modify?
As I understand, you left the values in the .env file to be the defaults, and then only modified the ports specific for nginx? Is it this part in the nginx section or something else?

ports:
      - "${HTTP_PORT:-80}:80"
      - "${HTTPS_PORT:-443}:443"

Thanks,
Nelson

[jniles]

Hi @nmambre,

I retained 80 and 443 in .env for the HTTP_PORT and HTTPS_PORT variables, respectively. However, I modified docker-compose.yml to use the non-standard ports I desired.

Here is my only change (generated using git diff):

I hope that helps!

[jniles]

Hi Dr. Gupta,

I didn't rebuild either service. As mentioned below, the only change made was to the docker-compose ports. This makes the proxy transparent to ODK Central.

Hope this helps!

[nmambre]

OK @jniles thank you for the explanation.
I made a similar change to the docker compose, and now get redirection errors, so I think my problem is now with to reverse proxy configuration. I'll have to read more about that.

[Dr_Vivek_Gupta]

So from what I can make out, the docker compose file is not using the .env configuration for nginx container.
Now in nginx container, the internal ports 80 has been mapped to host port 5050 and nginx internal port 443 has been mapped to the host port 5051.
This means I will need to proxy pass traffic from the upstream reverse proxy port 443 to the docker host up port 5050. I hope my reading is correct

[nmambre]

Did you check Configuring Upstream SSL?

This post helped me with the issues I had, maybe some is relevant for you.

ODK Central CI/CD (non default ports)

Hi Nelson, I was able to deploy ODK Central on Hetzner using Elestio. Here are the steps that I performed: 1- Update docker-compose.yml We need to update docker-compose.yml by following this instruction. So that it can read UPSTREAM_HTTPS_PORT environment variable. 2 - Create CI/CD pipeline Pipeline configurations: Build command: touch ./files/allow-postgres14-upgrade && git submodule update -i && docker compose build Run command: docker compose up -d With the above commands I don't ne…

[Dr_Vivek_Gupta]

Thanks Nmambre
My issue is resolved. It was to do with enketo container unable to reach my ODK central domain
I have detailed my steps at:

Enketo not loading

I have managed to RESOLVE The Error using various suggestions in the forum Step 1: Check reach ability of ODK host to the ODK central website https://xxx.yyyy.zzzz cd ~/central curl -i https://xxx.yyyy.zzzz curl https://xxx.yyyy.zzzz | grep getodk.org Potential issues: The server is host is not able to reach itself due to a. DNS issues: nslookup xxx.yyyy.zzzz You should get the public IP address of the ODK central website xxx.yyyy.zzzz If Not, check reachability to DNS and configura…

Best Wishes
Vivek