Self hosted ODK central stops repsonding

  1. What is the issue?
    We are self hosting ODK on Amazon EC2. It is not very busy as we are still developing our workflow. It occasionally locks up and stops responding. When it does this AWS usually but not always reports the server has failed 1/2 status checks. It may be that it always fails the check but it takes a while to show so I only see the error sometimes.

2.What steps can we take to reproduce this issue?
I am not able to identify any triggers for this. It occurs erratically.

What advice might you offer for trouble shooting this? What diagnostics should I be looking at?

M

1 Like
  • What version of Central are you running?
  • What EC2 instance are you using (e.g. t2.small)?
  • How much storage space do you have for the instance?
  • Are you running anything else on the same instance?
  • Have you customized your installation (e.g. by using a custom database server)?
1 Like

No ide why I dint give you that info before!!

versions:
894dc61c345edcc5998ab93a53e0d12ecc61a855 (v2023.1.0-2-g894dc61)
0b55d7d9f98f145c919c6e3476813275c4b69af7 client (v2023.1.0)
ad061f6088cf81c95bddf6f53310923cce9f76be server (v2023.1.0)

As we are just testing and developing currently its just t2.micro but with 30GB disk

No customisation, out of the box (was updated in Jan or Feb to what was then latest version form a version around a year old, all looked clean in upgrade)

Nothing else on instance

You could try adding swap: https://docs.getodk.org/central-install-digital-ocean/#adding-swap
But I think you'll need to setup monitoring to fully diagnose: https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/monitoring_ec2.html

Here's a possibly related post:

Apologies I can't be of more help.

1 Like

If status checks are failing, I'd stop and restart the machine. That usually moves it to more reliable hardware.

t2s are burstable instances so you might also be running out of CPU credit. Review your monitoring to confirm.

Finally, I agree with @danbjoseph that you should add 2 GB of swap. I'd probably also bump it up to a t2.small, but if you are price sensitive, looking at your current resource usage before making the switch.

All good points - will bump up to small. I think its CPU credits. Was being cheeky I guess thinking I could get away with micro. See if that solves it.

Thanks both

MUST NOT have server smaller than tx.small, else you will have issues. I learnt it after a lot of time and energy. Keep it on small or medium compute power, and you wont need swap disk.