Dealing with 25000 submissions on Aggregate

Sorry for inserting myself into a situation that is being resolved, but I want to ask this question, based on what I have read above ...
-> Chrissy, how are you dealing with the 25 000 submissions on the Aggregate server? Does it not slow down to being almost unuseable? It relates to a topic/request I made some time ago - having a method in Briefcase that would aid in deleting already pulled and processed records from Aggregate. The use case is the same - using scripts to do all the work of pulling and exporting from Aggregate, but then having to follow some manual process to clean up the Aggregate server.

Hi Mark,

Until recently this has not been a problem, but we are now getting some issues which I suppose are to do with large number of submissions.

I'm not keen on the idea of deleting previously downloaded instances (sometimes the XML files get messed up and re-download is a simple fix) but I do wonder if some kind of archiving system (for instance wrapping older submissions in to blocks of 100 to 1000, then letting briefcase pull these as a single unit rather than individually) would be useful.

I understand the apprehension about deleting previously downloaded instances and how this makes people feel uncomfortable. However, being able to clean up the Aggregate server programmatically is attractive - especially when the number of submissions is high or the size of each submission is large (I have the problem of both being true).

How do you currently keep the Aggregate server healthy? I mean - it cannot simply build up there forever - there has to be some clearing out sometime, right?

So the situation we are working in (I work with @chrissyhroberts) is a bit unusual perhaps in that we are assisting other international players in data collection in specific health situations.
The deployments are by nature time limited (i.e in 6 months we imagine the data collection is all done, we clear the server) but we can't really delete data in the interim, nor have a purely downloaded version because if we remove from the server then not all stakeholders can access it.