We run Central instances with much more data than you have that don't have this problem. The issue is most likely with your infrastructure.
Look at your logs. What do they say when you start a download with media?
You say you have no issue of RAM or CPU. What frequency are you measuring usage? Are you looking at the host only or also each container's stats?