Hello! Thank you for the response and your suggestions! These are very helpful.
We figured out that apache2 was now allowing the images through because the http response headers were not correct and but we couldn't figure out how to disable that 'protection' ('ProxyBadHeader ignore' did not work).
Today we ended up having tomcat directly work on port 443 with authbind and the problem is "gone". The web browsers are now able to load the images. To us this means the core problem was not apache or ajp... but something either tomcat/odk/db related.
As an experiment, I loaded up the developer tools in the browser with this working setup (just tomcat on 443) and looked at the raw headers when I clicked on the images in ODK Aggregate. They is still something funny with them... there is an extra ':' this time. The browsers seem more friendly to broken headers. Apache specifically had a module we couldn't figure how to turn off that would reject broken headers.
For the thumbnail image:
"Expires: : Tue, 25 Jul 2017 23:16:37 GMT
Last-Modified: : Thu, 13 Jul 2017 11:28:29 GMT
Content-Type: image/jpeg
Content-Length: 1676
Date: Tue, 25 Jul 2017 22:16:37 GMT"
For the actual image:
"Last-Modified: : Thu, 13 Jul 2017 11:28:29 GMT
Content-Type: image/jpeg
Content-Length: 2444370
Date: Tue, 25 Jul 2017 21:58:00 GMT"
We are just going to stick with this setup for now where tomcat is directly serving on 443 with authbind.
If we decide to again at some point try out apache or nginx proxy-ing to tomcat, we will start with a fresh DB and fresh surveys to see if this issue keeps showing up (we expect to flush our DB in the next few weeks anyways).
If you or anyone happens to have a little time and have a survey loaded in your DB with some images, can you see if the raw headers have some extra chars in the last-modified field?
Thank you so much again for the response! It was very helpful and we appreciate it very much!
Martin.