Pulldata won't work with larger csv files

Support
,
1

1. What is the problem? Be very detailed.
I am having an issue using the pulldata function with a larger csv files. Basically if the csv file is composed out of few rows, it works as expected but it fails with larger files. By failure I mean, no data is pulled from the csv in my form's calculated fields. I'll need to make something like this work with close to 20k rows in 5-10 columns. Is there a better way to achive such functionality, other than external csv file?

2. What app or server are you using and on what device and operating system? Include version numbers.
I could reproduce this with ODK 1.30 and 1.29.5 on Android 11 running on OnePlus 8

3. What you have you tried to fix the problem?
I have reduced the number of lines in the csv file and the form worked as expected.

4. What steps can we take to reproduce the problem?
My XML form and a csv file with fake data is in the attached archive. Open the form with the file as is, enter 1234567 and the other fields in beneficiary group won't be populated. Reduce the rows in the csv file to 20, load the form again, type 1234567 in the Search by ID field and the beneficiary group should be correctly populated.

5. Anything else we should know or have? If you have a test form or screenshots or logs, attach below.
sites.zip (37.3 KB)

2

We generally test with 50k+ rows on older devices. I've tried what you attached and the updates are instant on my Moto G5 running Android 7.1. Same with 5000 rows. Are you sure you're correctly getting the CSV to the device? Do you have another device you could try?

One thing I do notice about your test data is that you have ids that have leading 0s. I don't think those will match.

2 Likes
3

Hi @LN
Thank you very much for looking into this. What you've said really pointed me in the right direction and the issue is with my ODK Central setup.
To make the long story short, while adding media to forms, the ODK Central frontend will gzip any csv file that is larger than 16k. The NGINX, that I have as a reverse proxy in front of the ODK Central backend, was not configured properly to support gzip uploads, so the csv files were ending as gzipped blobs in the Collect app folder.

2 Likes