What is the problem? Please be detailed.
We are using slightly modified ODK aggregate to collect data using ODK Collect for very detailed forms (10 forms out of which the most detailed form has nearly 1400 questions). All was smooth in the testing phase, but with nearly 1000 field investigators submitting the forms, we have been observing "Error: Internal Server Error (500) randomly.
We have also been using Enketo to check the data being submitted for quality, and even the submissions through Enketo periodically time out with the same error on the web browser. In the case of the ODK Collect App, at least the app keeps trying to re send the data till it uploads, in case of Enketo it just fails and the modifications are lost. We are trying to modify the enketo code to keep a copy of the xml being uploaded (but that is not my query for this forum)
What ODK tool and version are you using? And on what device and operating system version?
We are using ODK version 1.4.15 installed on an 6 core machine with 6 GB RAM with tomcat 7 (documentation says to install tomcat 8 with versions > 1.4) and openJDK. I have keeping a constant watch on the resources on the server and they don't seem to saturate even when multiple users face the error at the same time.
What steps can we take to reproduce the problem?
I can share the structure of the forms separately if needed.
Anything else we should know or have? If you have a test form or screenshots or logs, attach here.
The smaller forms all upload smoothly. We also faced problems uploading some of the bigger forms through enketo with 200 "Payload too large error" that we fixed by increasing the upload limit from enketo code from 50 KB to 2 MB.
What can we do to improve the upload performance on the aggregate? Will increasing RAM/no. of cores help? Please help me find out the choke point.
Thanks!