I self host using UK based data centres and have made choices based on their published 'credentials' about (for example) energy use, rather than lowest cash-cost or highest performance.
I have a 60GB cloud server (low end) with a further 60GB of attached storage and use BorgBackUp to manage incremental backups. My server has been live for 3 years. No down-time, all updates / upgrades of Central have been hassle free and simple by following the docs. I don't always upgrade Central immediately, but usually within a few weeks when there is a shiny new toy included 
Last year, ahead of an anticipated big data collection exercise, I took the plunge and picked up 1TB S3 storage from peasoup.cloud (also UK based data centre). I have created some private buckets for Central external storage (media / attachments) and my backups. This costs me £5/month as a fixed fee (i.e. no ingress / egress / API fees). The 60GB of attached storage on the cloud server costs the same, and I'm thinking of dropping that as it has been unused since switching to S3. My cloud server is with a different provider.
So far I have had no problems with performance and have gone from 87%+ disc full to 35% (and remaining around that level now that the attachments are stored in S3). Some of the space was recovered by deleting old docker images, but most was from attachments. I didn't monitor exactly when disc space was recovered - the transfer of attachments was done in stages (I think), but I can see the audit logs transfer attachments on a daily basis to S3 and the database size doesn't change massively now (if I check when I'm doing server maintenance).
I have had a few times where form updates on ODK Collect have failed (or new test forms have not loaded immediately) but these are not persistent nor have any recognisable pattern, so I can't 'pin' it on S3. More likely my wifi or device perhaps?
I retrieve all my data via API using QuODK - there is absolutely no change since moving to S3. Likewise in the Central UI / Enketo (edit submission) this works seamlessly. You see no difference as a web user / administrator.
So I don't have to worry about this at a practical level - I think this actually means that Central is encoding the data so it can read it uniquely rather than 'exporting' the files with their human-readable name (which makes sense to me if a large data gathering exercise could theoretically produce images with the same name based on the timestamp, or more likely use the same names for media files in different projects / forms).
I suspect that some folk were hoping to be able to access files directly from S3, so this is probably just a caveat that you can't. To be fair, I think that the ODK team are very careful not to over-promise things, so tend to add notes of caution about (potential) 'limitations', but in every case I have found that Central performs flawlessly. However, anyone hoping to get under the bonnet / hood might not find things easy to tinker with... I'm OK with that.
In summary, it has been a positive experience moving to S3 and I am reassured that the database size does not expand exponentially with data collection exercises... I have not gone with the 'bigTech' providers and have not experienced issues.