Hi,
I have a huge ODK database, containing around 80k form submissions going upto the size of 136GB (database size). I have kept an eye for its health, but now I suppose it is touching it's tolerance limits. For now, I am unable to export the data into a CSV. It was working till around 74k submissions yesterday. In tomcat logs, I am getting following error:
INFO: iteration 740 before issuing query for beneficiary_reg_form
Jul 22, 2020 5:11:52 AM org.opendatakit.aggregate.task.CsvWorkerImpl generateCsv
INFO: iteration 740 before emitting csv for beneficiary_reg_form
Jul 22, 2020 5:11:53 AM org.opendatakit.aggregate.task.CsvWorkerImpl generateCsv
INFO: iteration 741 before issuing query for beneficiary_reg_form
Jul 22, 2020 5:11:53 AM org.opendatakit.aggregate.task.CsvWorkerImpl generateCsv
INFO: iteration 741 before emitting csv for beneficiary_reg_form
Jul 22, 2020 5:11:53 AM org.apache.commons.dbcp2.SwallowedExceptionLogger onSwallowException
WARNING: An internal object pool swallowed an Exception.
org.apache.commons.dbcp2.LifetimeExceededException: The lifetime of the connection [590,085] milliseconds exceeds the maximum permitted value of [590,000] milliseconds
at org.apache.commons.dbcp2.PoolableConnectionFactory.validateLifetime(PoolableConnectionFactory.java:424)
at org.apache.commons.dbcp2.PoolableConnectionFactory.passivateObject(PoolableConnectionFactory.java:364)
at org.apache.commons.pool2.impl.GenericObjectPool.returnObject(GenericObjectPool.java:578)
at org.apache.commons.dbcp2.PoolableConnection.close(PoolableConnection.java:206)
...
...
...
...
Jul 22, 2020 5:12:28 AM org.opendatakit.aggregate.task.CsvWorkerImpl generateCsv
INFO: iteration 782 before issuing query for beneficiary_reg_form
Jul 22, 2020 5:12:28 AM org.opendatakit.aggregate.task.CsvWorkerImpl generateCsv
INFO: wrapping up csv generation for beneficiary_reg_form
Jul 22, 2020 5:12:28 AM org.opendatakit.aggregate.task.CsvWorkerImpl generateCsv
WARNING: stale CSV activity - do not save file in PersistentResults table for beneficiary_reg_form
I assume its hitting some kind of timeout value for tomcat, since I did upgrade the cloud machine to a higher power, but the issue remains the same.
Any idea how to handle this?
Thanks,
Saad