Possible memory leak in ODK Central backend

Hello,

We are observing the memory usage of the Central's node processes increasing over time, to the point where the entire server where ODK Central backend is running crashes.

We run ODK Central using the docker images on a GCP Compute Engine VM (e2-medium) running Linux Debian 10.

To mitigate this problem, we scheduled the server to restart every day, so we free up memory before the system crashes. But of course, this is not an optimal solution.

We are not sure where exactly the problem is, but we noticed that this memory leak started when we started to pool the Central server for submissions, using the OData endpoints, so we can import those submissions to our databases to process them further. We pool the Central server every minute.

Is anybody having the same problem, or have any idea how to solve it? Please let me know if you need more information.

Thanks in advance.

Best regards,
Marcelo

1 Like

What version of Central are you running? If it’s not the latest (v1.2.x) can you reproduce it on the latest version?

1 Like

We are running version 1.2.0.

We are going to update to v1.2.2 and let you know if the problem persists.

Thanks.

1 Like

If you are running 1.2.0, upgrading to 1.2.2 won’t make a difference. My guess is that the polling every minute is the issue. That feels excessive to me. Increase to 15 minutes and see if that helps.

Thanks.

I apologise for the delay in the response.

We increased the interval between calls from 1 minute to 5 minutes. We are observing to see if the memory leak persists. I'll report back here after we have a conclusion.

Kind regards,
Marcelo

Hi @yanokwa !
I can't follow the logic here: If there is a memory leak, it doesn't matter lowering the load to the service. It will it only delay the OutOfMemory...

We've increased the interval and as you can see, the memory consumption is still rising...

Hi @yanokwa! It would be nice if this could be acknowledged as bug.

The solution-marked post is not a solution, only a delay for this issue. Could you at least remove this marker?

I agree that there is a likely bug. No ETA on when we can look into this, but if you have an hints as to what is driving this (e.g., exports, submissions, form conversions), that'd help move this along a little faster.

1 Like