ODK Central memory issues

1. What is the problem? Be very detailed.
Attempts to "Download all 300 records" from ODK Central failed repeatedly, although an earlier download of 15 records had worked fine. An attempt to log on to the Digital Ocean console raised an insufficient memory error. I increased the memory from 2GB/25GB disk to 3GB/25GB disk, which solved the immediate download problem. However, we expect a total of perhaps 2400 completed surveys over the next few months. and need guidance on how to configure the server to accommodate these surveys.

2. What app or server are you using and on what device and operating system? Include version numbers.
ODK Central v0.7.1 on Digital Ocean droplet

3. What you have you tried to fix the problem?
As noted, increasing memory to 3GB solves the immediate problem, but may not be enough for 8X the number of submissions.

4. What steps can we take to reproduce the problem?

5. Anything else we should know or have? If you have a test form or screenshots or logs, attach below.
Survey size, based on downloaded csv data file: 146 columns, including columns added by ODK
Includes one photo, "Very small" image size
Size of downloaded zip file with 300 survey records: 518,403 KB
Projected size of zip file with 2400 records: ~4GB
Number of surveyors: 22; each is responsible for 100+ surveys
(I also submitted a feature request to allow partial downloads. This would be useful, but I'm not sure it would solve the insufficient memory problem.)

Thanks for providing details about your scenario. Could you also please share roughly how many questions are in the form definition?

1gb of memory is tight for exports with images so I'm not surprised you've hit this issue. However, since you have small images, I'm guessing that you probably don't need much more that 1gb, even for a large number of records. From our tests, I believe 2gb should be plenty.

Since it's just the export that has a high memory need, there's should not be much risk in underestimating memory and adding more as needed. I would go for 2gb or 3gb but if you're really looking to keep the box as lean as possible and have server management experience, you could look at the tip in the documentation about adding swap.

Could you please make sure you've posted it? I'm not seeing it!

Another option would be to use Briefcase to pull your submissions. You can likely do this with 1gb of RAM and no swap but you'd have to check.

LN - Thanks for your response. I'm happy to learn that these download problems do not risk corruption or loss of data on the server. Here are some additional comments.
o The number of questions on the form would be the 146 columns in the csv data file, less the 8 columns added by odk, = 138 questions. This includes hidden questions (calculate, etc.) and questions without data (not-relevant, note, etc.). The form has four languages.
o Essentially all of the zip file consists of the jpg photo files, which of course are already compressed. For the latest download, the compressed csv file was 0.01% of the total zip file. This is using the very smallest image size.
o The memory requirements you cite may be optimistic. I have been using 2GB from the beginning, since the 1GB suggested in the documentation was insufficient to install Central. 3GB works for now, but will lilkey be insufficient for long, as more surveys are uploaded. I will try the swap as suggested.
o The suggestion for partial downloads was submitted on the page I found when looking for an updated release that might solve my problem. This is the page:

Is there a better place to submit such requests?
Regards, Hayden Boyd

Thanks for the additional details! I had missed that you started from 2GB.

That's good to know and surprising. Do you remember at what part of the installation process the failure happened with 1GB?

That makes sense! I was looking for it at https://forum.getodk.org/c/features/9 but I think the comment provides sufficient detail.

re installation failure with 1GB. Sorry, I do not recall the exact point of the failure. The original installation was an earlier release, and I upgraded to v0.7 some time last Dec-Jan.

re adding swap. The first line of code is "fallocate -l 1G /swap" . Since I am running 3G memory, should I change the 1G to 3G?
Thanks for your help!

I chatted about this with @Matthew_White who reminded me that you'd previously helped us fix a Central export bug (ODK Central corrupts large csv media files). There were a couple of versions around that time which had higher memory needs on build. We have since addressed those and we verified that 0.8.x can be built on a 1gb droplet.

This seems large for images set to "very small". Could you please let us know the size of one image in KB and pixels?

The 1G is how much swap you'd like to allocate. I would recommend starting with 1G. Swap is slow so ideally it's only used rarely for operations that temporarily spike up memory need.

We have recently found that one of the libraries we use has a slow memory leak. We don't think it is experienced often but since you've been running this server for a while, it could be a possible explanation for your higher-than-expected memory needs. Have you restarted Central with systemctl restart docker-compose@central recently? If not, it might be something to try. If you do try it and find that you are no longer running into memory issues, that would be good to know.

LN - thanks for your response. Image size is indeed a problem. It turns out that some, but not all, of the Androids that have submitted data so far do not have image size in ODK Collect set to "Very small." This is puzzling, since all of them were set up by copying the settings QR code from a master device. My colleague in Malawi is contacting surveyors to check the image settings on their devices. If I run into memory problems in the future, I will enable swap. Meanwhile, I've upgraded Central to v0.8.1. Although I already restarted the DigitalOcean droplet after upgrading the memory, just in case, I also restarted Central as you suggested. ODK keeps getting better and better, and Central is wonderful. Thanks to all the ODK team. Hayden Boyd