2. What app or server are you using and on what device and operating system? Include version numbers.
ODK Central versions - @Thalie?
3. What you have you tried to fix the problem?
Currently triaging.
ruODK tests run against a production server, so I'm not immensely keen to crash it on purpose.
4. What steps can we take to reproduce the problem?
Thalie reports this behaviour from ruODK::submission_export being called with an incorrect passphrase against an encrypted form.
5. Anything else we should know or have? If you have a test form or screenshots or logs, attach below.
I'd expect:
ODK Central not to crash
ODK Central to return an informative http status and error message when the passphrase is incorrect - unless this is a security risk (brute force attacks?)
DigitalOcean 4 GB Memory
client v1.1.2
server v1.1.0-7-ga33bc6f (not sure why not v1.1.1 as the other ODK Central instances I am using... as I thought I had done all required updates - I may have to check this into more details...)
I have just checked the query outside of ruODK and get code 400.12 - Could not perform decryption. Double check your passphrase and your data and try again.
I was not very precise when I said "crash", the web interface is still online, but you can no longer visualise content (e.g., projects), the backend becomes unresponsive, requests do not work any more, and the services need to be restarted.
The behaviour is not absolutely reproducible (I had to run the request 3-4 times to get the server unresponsive), so I would suspect timeout due to overwhelming memory resources? (sorry servers are really out of my field)
Is it possible to stop processing the request early if passphrase incorrect to preserve server memory? Thalie's log indicates the db connection pool was exhausted.
What's the most informative server status to return - 500 or something in the 400 ranges like @Thalie suggested? My ODK Central 1.1 returns HTTP 500, Thalie's Central returns 400.12 (nice - why am I not getting a 400.12?).
@LN even a single request with an incorrect passphrase can crash my ODK Central server (v1.1) when sent from sufficiently many different test environments.
I'd like to file this here (for visibility) as a bug report against ODK Central's backend: prevent submission_export requests against encrypted forms with an incorrect passphrase from consuming too much server memory, and return an appropriate HTTP status (if that's not already done).
The 400.12 is what's returned by Central which I think is good and expected. I think you probably should terminate requests on that as well since it's not recoverable. The 500 is likely because you got to the point of reaching the KnexTimeoutError which @Thalie shows in her screenshot.
Chatted briefly with @issa about this and it does seem like there's a database connection leak though Central code does seem to free connections when it should. It doesn't appear to be related to memory. First thing will be to see whether it's reproducible on v1.2 where the database layer has been switched out (in part because you experienced an issue we thought was fixed by @issa's patch to the database library).
The issue is still there with Central 1.2. I have just encountered it again after (very stupidly) passing several times the same (wrong) passphrase to ruODK.
Sorry to hear that and thanks for reporting it, @Thalie. If you or someone else gets a chance to review logs, it would be helpful to see what the new error looks like (we removed knex in v1.2 so it will not be quite the same).
I guess I unconsciously wanted to check this! Since it happened on a production server on which I am not admin, and the IT team is not be super available this week, I will replicate the procedure on my test server and export the log (probably 2nd part of the week).
@LN I have eventually found some timeframe for testing
ODK Central config:
client (v1.2.2)
server (v1.2.1)
After 3 to 4 unsuccessful attempts passing an incorrect passphrase to ruODK, here is the corresponding log with a ConnectionError: timeout exceeded when trying to connect
Hi Issa, yes all the forms have client audit logs and some of them also have media attachments that are usually downloaded with the export (although I may change this as the database is getting bigger and a daily export of the media attachments is not absolutely necessary)