I think it will time out (as it does for me on 1,500 submissions without attachments).
Approaches I can think of
Still using briefcase
Generate file in background and provide link to it - have seen this done on other platforms when the file size is too big for a standard immediate download to work
Purging successfully extracted submissions might be the way to go there.
I definitely don't think you want to permanently purge exported submissions from central - this would not be ok from a Good Clinical Practice perspective for medical/clinical research studies - you need to maintain the original database/records. For example some of our aggregate servers have data on >100,000 individuals in a study.
Might work for some people but doesnt work well for us in clinical studies where we want people to practice using the forms exactly as they will in real life / hospital etc.
Maybe we just have 2 versions of form where one has ID XYZ_training, but personally I still think a purge submissions function could be useful
I think it will time out (as it does for me on 1,500 submissions without attachments).
it shouldn't. central is architected to continually stream data out as it is processed, which under normal conditions should not cause a timeout. if something else is happening in your setup, then either we have done something we did not intend on the server side to prevent our designed behaviour, or else the additional layers present in your environment are causing problems.
as for questions around draft or training forms, we are aware that it is a big pain point and we will have some kind of answer to that concern at some point. there are a lot of other smaller questions that will need to be gradually answered before we can get there.
@issa re: the timeout @dr_michaelmarks reported
Smaller forms (around 200 entries - zip 90 KB) download just fine. The larger form (12 fields, 1200 entries) throws a timeout error on the console
Unqualified comment from the sidelines:
Is the ODK Central that times out behind another proxy with its own timeout settings?
What's the setup and deployment?
Are there logs? (even inside containers e.g. written to /var/log/odk/std_*)
@Florian_May Looks like the Nginx proxy is timing out trying to retrieve from Central.
Logs in the service_1 container don't suggest anything wrong -
Nothing in /var/log/odk/stderr.log and only ::ffff:172.19.0.5 - - [11/Sep/2019:10:21:36 +0000] "GET /v1/projects/6/forms/baseline_hh_survey/submissions.csv.zip HTTP/1.0" 200 -
in stdout.log
The QR codes need to have a human readable component
I.e
Project name
App Username
This is really important when a supervisor is setting up lots of devices (especially when doing it remotely in a different country from the main IT support team) so they know each device is being configured with the correct settings.
This will become even more important once it is possible to setup different app users with different form access on a given project.
Related issue is that it's very difficult to look at the subsequent URL in the settings and make sense of it - I.e easily know which server / project you are linking too - this is important when we have field superviaors overseas and we need them to be able to easily check that they are connected to correct project / server
It's not only collect though - the actual QR code image central generates under that would ideally have some text that says
URL
Project name/number
App User Name
Also an option in Central to download / email the QR code would be very very valuable to help remote users setup devices
(Ideally automatically naming the QR code file based on Project name / App user-nickname)
If I understand correctly, @dr_michaelmarks, you'd like the image that includes the QR code to also include identifying information because it might be emailed, printed out, etc, is that right? That is, there's sufficient context for the QR code when it's displayed from Central but a supervisor setting up devices would not be logged in to Central.
I also do agree that it should be possible to review the server/project/app user configuration from clients. The project id is listed at the end of the URL but it's just a numeric identifier. Perhaps as a first step the project name could be appended to the URL (sluggified) and ignored by the server? For username, I think that if the QR code json included the app user name associated with the username key, the username would show up in the server settings.
Correct - this is very common. See also my suggestion above building into central a download/email button for the QR code.
Wanted something like my example above:
Happy to discuss how to improve the display of URL within Collect; like your suggestion @LN to include project name. If we could something similar for App User Nickname that would help.
okay, i understand more about the QR code now. i didn't have the context that people were downloading the image as its own artifact; we can probably come up with a different solution there that achieves your eventual goal.
for background, from our research we'd concluded that most deployments involved the phones being set up ahead of time by a small centralized staff who could just provision the phones directly from the panel. being able to send provisioning information elsewhere was something we'd entertained but eventually ruled out because it sounded like it wouldn't be as useful. we can come up with a solution that's tailored for that approach rather than try to shoehorn things into the existing process.
as for the collect display of information, some of these proposed hacks might work, but i think some of them will cause problems with how collect authenticates with central.
okay i looked into the timeout issue. turns out the server was doing the right thing and i neglected to configure nginx to match. silly and simple problem. silly and simple fix:
(the increase to a 2 minute timeout is probably not strictly necessary. it's a per-chunk timeout, so it will only trip if no new data is streamed back to the client within that window. i artificially inflated row-processing time by 50ms per row and the first chunk came back at 58 seconds so i figured 2 minutes wasn't a terrible idea for some diabolically complex forms, possibly. i don't see the harm in increasing by 60 seconds.)
@dr_michaelmarks@issa I added the extra lines from the diff into odk.conf but the download is still failing with a Network error. Only now after 120 seconds.