ODK Central v1.2 - Primary data table (.csv) download

Support
1

1. What is the problem? Be very detailed.

I cannot download data from an encrypted project when selecting the Primary data table (.csv) export. ~ 550 submissions / 322 columns

image

image

Both All data and media files (.zip) and All data without media files (.zip) however work fine and ruODK functions that rely on the RESTful API retrieve the complete data set without any problem.

2. What app or server are you using and on what device and operating system? Include version numbers.

ODK Central
client (v1.2.1)
server (v1.2.1)

3. What you have you tried to fix the problem?

Submissions from some other forms in the same or other (encrypted / non-encrypted) projects can be downloaded correctly, but none of them has reached the same number of submissions yet so I cannot fully assess this yet, but may be able to do so some time next week.

Interestingly, when filtering submissions for a few days only, the CSV download works again but stops working when above ~100 submissions. I have observed exactly the same behaviour with approximately the same cutoff with another form that only has 84 columns.

I will still continue to check if this could somehow be related to some of the content collected, but at this stage, I would rather lean towards a possible bug?

4. What steps can we take to reproduce the problem?
See description above

5. Anything else we should know or have? If you have a test form or screenshots or logs, attach below.

I requested the server log from the admin, will share as soon as soon as I can get my hands on it.

1 Like
2

Decryption and writing the CSV are both resource-intensive and that could be the source of the issue. What CPU/RAM does your server have?

3

OK - I would have expected the generation of the *.zip archives to require more resources than the generation of the CSV alone. Any reason why the former would work, while the latter would not?

If this is a resource issue, this is not good news since the main trial will run will the same (encrypted) project configuration and is expected to host 110,000 submissions... The initial server requirements were 4 cores / 12 GB RAM, so I would not expect resource issues at this stage, but it also seems we have recurrent issues with this specific server. I will be in-country in ~3 weeks and will investigate more at that time. Because we are starting piloting in another country (on a server that is fully dedicated to the project), I will also have a comparator.

4

It seems I missed this part, apologies. If it’s limited to the CSV export, this does indeed seem like a bug. We will look further into it.

That sounds very reasonable. Between this and the fact that other export variants work, it doesn’t seem like a resource issue after all.

5

My bad, I realise that it was a bit hidden in my post.

I would not totally exclude an issue with the way these hardware resources are used. We encountered memory issues with this server already, which may not be totally solved + It is a shared resource and I am not sure exactly how many projects are actively running on this ODK Central and what is the volume of data it manages, while we have dedicated servers in other countries.

1 Like
6

Hi @Thalie! I wanted to ask, what happens when you try the "Primary data table (.csv)" export? Is an error message shown in the browser? Also, were you able to access the server log?

7

Hi @Matthew_White, there is no error message, only the download never starts after having been requested in the Decrypt and Download window (even when trying several times as suggested).

image

image

I have followed up with the IT admin for the exact log when requesting the download, waiting for his answer.

For comparison, it takes ~3s to start the download when selecting All data without media files (*.zip) instead

1 Like