I have been able to reproduce the problem and I can confirm the issue is the external mapping of the port.
It's unlikely that support for non 80/443 ports will be added soon, but I've filed an issue at https://github.com/getodk/central/issues/138 so the Central team can discuss it and perhaps add it to the roadmap. Would you be open to sending in a PR?
As an alternative, you could also try setting up a reverse proxy on your machine and routing port 80/443 to the existing server and routing 4480/4443 to Central.