Forms not downloading or rendering on reverse-proxied, tunneled ODK Central server

1. What is the issue? Please be detailed.
I have an ODK Central instance hosted on a machine behind a NAT. I've used Rathole to tunnel the ODK Central instance to a public IP (hosted on Digital Ocean) and everything works normally (DNS, form upload, login, project listing, email). However, when trying to preview forms or downloading on an Android device, I get "Could not connect with Form Server" and "Form download failed", respectively.

I have followed the debugging steps for DNS in https://docs.getodk.org/central-troubleshooting/#preview-could-not-connect-with-server to no success. I also can download the compiled XML, which indicates that the problem wasn't a silent fail in the form upgrade. When checking the ODK logs, the nginx server returns a "POST /-/transform/xform/[id string] HTTP/1.1" 500 error, and I cannot find more detailed logs than this.

Finally, because I'm using a tunnel through a reverse proxy, I have the "upstream" SSL configuration in the .env file, with both ports defined for HTTP and HTTPS.

I'm running ODK Central version 2023.4, the latest ODK Collect, and hosted on Debian 12 via docker compose.

2. What steps can we take to reproduce this issue?
You would need to set up a reverse proxy and tunnel to a local server that is running the docker-compose version of ODK Central, as I have described above.

3. What have you tried to fix the issue?
As mentioned above, I have tried to look into the logs, and I also followed the DNS debugging steps, thinking that it was a DNS issue.

4. Upload any forms or screenshots you can share publicly below.

I'm pretty savvy with Linux - how can I get better logs to see what might be causing the 500 failure? Is there an easy way to get those logs? I've looked through the enkento, service, and nginx containers to try and find a more detailed error message, but could not find any.

Hi @jniles,

I'm not totally sure about your situation but I'll write some thoughts that I hope are useful clues and don't lead you astray...

  1. POST /-/transform/xform/[id string] is about Central communicating with Enketo (- is a proxy to the enketo service running in its own container). I expect that 500 error to occur on form draft upload or publish. You might be able to look in that container to get more details about that particular 500 error, or enketo may not be running/available. If you can't preview forms through the Central website, that's another clue that something isn't set up right with enketo.

  2. Not being able to download forms on Android is a separate problem and shouldn't be related to that 500 error. Collect uses the Central OpenRosa API over HTTPS to get the form lists/forms, so if it's not reaching the Central server, it could be an issue with ports and/or certificates. Knowing that, if you look up your errors in the rest of the ODK forum, you might find more clues, or at least other people also having a trouble running Central outside the normal (recommended) 1 machine + 1 public IP setup.

Thanks for the reply @ktuite! It gives me somewhere to start digging. I'm suspecting somewhere the PORTs are either being used in a way that conflicts with my port-forwarding schemes. I'll do some digging and maybe ping back when I have more details or questions.

Thanks!

Thanks @ktuite, this pointed me in the right direction. I now have the OpenRosa API correctly resolving forms to ODK Collect. I hadn't realized that it would inject the $PORT if the port was non-standard, but it makes sense that it would (e.g., the form list is resolves as $DOMAIN:$HTTPS_PORT/v1/...) from Collect.

If anyone is also running behind a reverse proxy, the way I solved this issue was to maintain the $HTTPS_PORT and $HTTP_PORT as 443 and 80 respectively, and manually update the docker-compose.yml to use my own non-standard ports (e.g. 8443, 8080). Collect looks for the forms at $DOMAIN, and my reverse proxy can handle the rest.

I'm still debugging the Enkento issue, but I don't have a use-case for Enkento at this time... it's just nice to have.

EDIT: Enkento is now fixed! I think it required a docker rebuild, and likely suffered from the same issue. :muscle:

Thanks so much @ktuite!

1 Like

Hi Niles
I am in a similar situation
Can you share the relevant portions of ,env and docker-compose for enketo please.
How did you rebuild enketo only? Did yo also have to rebuild redis ?
Thanks
Vivek

Hi @jniles
Which portions of the docker compose did you modify?
As I understand, you left the values in the .env file to be the defaults, and then only modified the ports specific for nginx? Is it this part in the nginx section or something else?

ports:
      - "${HTTP_PORT:-80}:80"
      - "${HTTPS_PORT:-443}:443"

Thanks,
Nelson

Hi @nmambre,

I retained 80 and 443 in .env for the HTTP_PORT and HTTPS_PORT variables, respectively. However, I modified docker-compose.yml to use the non-standard ports I desired.

Here is my only change (generated using git diff):

I hope that helps!

1 Like

Hi Dr. Gupta,

I didn't rebuild either service. As mentioned below, the only change made was to the docker-compose ports. This makes the proxy transparent to ODK Central.

Hope this helps!

1 Like

OK @jniles thank you for the explanation.
I made a similar change to the docker compose, and now get redirection errors, so I think my problem is now with to reverse proxy configuration. I'll have to read more about that.