Hi @LN
Thanks as always for your considered view of your end users changing needs and our feedback - just when you think you've solved a problem...
I recognise the difficulties of pleasing all the people all the time and finding a 'right' answer.
In my case, of collecting and managing data from a relatively small number of enumerators, I have always worked on the basis of "once it's sent, it is no longer possible to edit", but until that point, you can do what you need to. In general I have not differentiated between finalised and draft except as an advisory where the form needs to be updated as part of the data collection process (i.e. all answers cannot be given in a single session such as location at the end, or the distance travelled etc) - usually it is easier to find the draft submission than a finalised form. To date I have been able to leave that to the enumerator to decide, with the caveat that they need to turn off data to avoid early submission.
Personally, I cannot see a helpful situation where an enumerator can edit SENT data as this has plenty of opportunity for conflict - I generally analyse data after downloading it from Central (incrementally), so that would introduce an additional risk if there is a possibility that the existing records could change and no control over when that might happen. And then there is the issue of 'delete after send' (opt in). That situation would mean that enumerators would not have the opportunity to edit sent data because it would no longer be on the device...
For clarity I have no 'authenticity of data' issues to deal with and for some surveys it is actually beneficial to review the data before sending (when it has been snowing and the notes are very cryptic, for example) or as I highlighted when additional information comes to light. I understand that others may have reasons to prevent editing of 'finalised' forms - but presumably they would also need to have a way of understanding what, where and when questions have been edited after the original survey session.
The 'introduced complexity' of managing offline data and entities adds whole dimensions!
Thank you for looking at our form. I could not send a form without an accompanying explanation. Since the explanation is quite extensive, I have made a showcase of it. you can find the showcase and one of our 78 forms here:
I am interested to know if you can make Enketo open a finalized form correctly.
When I tried it (on Kobotoolobox), a lot went wrong, starting with not handling the cascade choicelists...
It really depends. Since 2023.2 came out, we have been actively watching translations come in and have scheduled releases just to get more translations in because we knew the new and changed strings were disruptive. Typically we only update translations when we already have a release planned or when a community member specifically requests a release. There are times where this would be hard to accommodate but so far we have always been able to honor those requests.
This is really great to hear, thank you for that feedback. Another option that you could consider is to leave the location question as non-required. If it is left blank, you could ask a question like "The location was left blank. If you need to send anyway, explain why." This text field could be required. Then the user could fill in an explanation if something unusual happened like their phone wasn't getting a location at all. If they don't supply an explanation, for example because they just need to step outside and try again, the form would not be finalize-able so they would have to save as draft. There are lots of variations on this approach including using one or many select questions instead of an open text box.
I see, that makes a lot of sense and explains why staying on 2023.1 is a better option for you at this time. We will consider that as we think of longer-term options.
Thank you, these are on our radar. We don't have a timeline for them yet but we know that they're important.
Yes, this has potential! There are interesting questions around what the experience should be when finalization fails and around how bulk select should be accessed.
Thank you so much, this provides so much valuable detail!
I see that you make use of the search() appearance function. This is indeed not supported by Enketo and also not included in ODK docs (though it's in xlsform.org and still maintained for now). The alternative functionality that works in both clients is select_one_from_file.
The only downside is that the performance for select_one_from_file hasn't fully been optimized yet and it can become slow for very large attached datasets or complex queries. I see you have 10k items in one CSV and 35k in the other. I haven't tried your specific form yet but we generally test with 60k+ items with no performance impact on modern devices. That could be an option to consider if you do want to try server-based edits. We also maintain Enketo and have made several performance improvements recently which you can try out at https://getodk.org/xlsform/ (Kobo and others that use ODK clients often run older versions).
It sounds like your scenario is quite similar to @seewhy's. Your data collectors are motivated and trusted, the data they are collecting is largely objective, and the thing that you're collecting data about is largely unchanging in a short time frame (as opposed to e.g. someone's health). It is possible to make honest mistakes and then it is desired to revisit the environment and fix those errors.
I realized that unfortunately you won't be able to use https://getodk.org/xlsform/ to verify the very latest speed improvements because it doesn't support attachments. But it will be worth trying with select_one_from_file on older versions too.
Thanks again to everyone who has taken the time to describe their workflows and needs.
We have done some iteration and are currently planning to add bulk finalization of drafts as suggested above and a check for errors at the end of the form filling experience.
Here is what we have in mind for bulk finalization:
It's an all or nothing action: you can't apply it to a subset of drafts. This is because we don't want to add complexity or visual clutter for most users. If there are forms that aren't complete and definitely need to be edited, the form design should include a way to indicate that so that finalization can be blocked. Any forms that can't be finalized will stay in drafts and will be indicated as having errors.
The functionality is in the overflow menu rather than directly on the app bar. This is because we expect it to be a relatively infrequent action, even for a project in which it's a key part of the workflow.
Bulk finalization will show a progress dialog and block the user from using Collect rather than working in the background. We don't think there's anything useful to do in Collect in parallel and doing this in the background would increase complexity and risk.
Another thing we will do is introduce an automatic check for errors at the end of the form filling experience. Currently, this only happens when a user attempts to Finalize/Send. This can be problematic if a user saves every form as draft and later bulk finalizes. If they accidentally skipped some questions and save as draft, they wouldn't find out about the missing answers until bulk finalizing when they might not have access to whatever they were collecting data about. So we think that to make this flow complete we also need to provide earlier feedback about form errors.
In the new Collect version with the features above, we would recommend that people who used to always finalize and then make edits as needed migrate to always saving as draft and then bulk finalizing as needed. They can hide the "Finalize/Send" button at the end of form filling to guarantee going through bulk finalization.
Hi LN,
Why did you choose for a bulk finalize which is a all-or-nothing choise?
Is it not possible to give the user select/deselect boxes for every draft form and buttons to select/deselect all? and a button "finalize all selected drafts"?
This exists now in the ODK-collect 2023-1 version under the "send finalized forms":
There one can choose which form to send/ or select-deselect all and then bulk-send all selected forms.
Why do I suggest this: In our process it happens that we would have a few forms that we want to keep in draft stage because we need to go back to the vegetation plot and determine species. In such case we would not be able to send any form because we only can go through the bulk-finalize and send process if ALL forms are really finished.
So we would have the risk of keeping a lot of data on our mobile device, just because of some remaining draft-forms.
If the species is required and not filled in, those submissions would not be marked as finalized. Only forms that are complete and have no validation errors are eligible for finalization. What we recommend in a case like that if there isn't already a question that is required and left blank would be to add an extra question at the end that would be something like "Have you fully verified every answer?" and if left blank or marked as no, finalization would be blocked.
To be explicit, if you need to selectively make sure that certain filled forms aren't finalized, you could open each of those, edit them such that they can't be finalized (e.g. answering 'no' to "Have you fully verified every answer?"), and then bulk finalize. It is higher-effort than selecting all and unselecting the forms you want to skip but we think it's the right tradeoff given that this kind of workflow is less common than others.
Dear @LN,
here another workflow which unfortunately will no more work now:
We used the finalised status to mark the cases for local review with/from a supervisor before sending them. The supervisor will fill few (mandatory) fields QA status, her/his id & secret password, optional remark - before the form can be sent by an enumerator (or supervisor).
So, our requirement is to have a status (from the enumerator) which marks a "finalised" case for review by/with a supervisor before a form is / can be sent. (A bit similar to the validation status in Enketo.) If the QA finds problems, the enumerator could even go back to the household to fix the data. Finalised status can be removed (and reset).
Our experience and recommendation is to allow quality assurance directly in the field, still during data collection, esp. in remote areas and with teams moving around (and with limited workflow support for editing/sending back individual submissions.)
Another use case for the (old) finalisation was that an enumerator or supervisor could at any time press "finalisation" to run all checks (using move to the end). This was even helpful to position to a last edit point (e.g. mandatory question) when an interview was interrupted and reopened.
As far as we understood your "non-edit" change was mainly caused by the new "entity" concept. I am not sure if this was the best idea. As several postings indicate, "finalisation" seems another, separate concept and an established workflow element.
Additionally, we want to keep the Collect settings option (also for Central), please: set auto-send off/on, set automatic finalisation off/on.
Thanks for adding to the conversation, @wroos. Workflows with in-field review were a significant motivator for reworking the way finalization worked. Previously, because finalization was not actually final, if a supervisor reviewed and signed off on a submission, unless they could send right away, they had no way of ensuring that a data collector didn't go back and make further edits. This is one of the many problems our user research identified with allowing submissions to go back and forth between draft and finalized states.
Here is how we expect workflows with a supervisory step to work:
The form has questions intended for a supervisor (as you described)
When a data collector does their work, they can't finalize because they end up with missing required questions for the supervisory process
If there's a need to quickly differentiate between drafts that need some data collector action vs. drafts that just need sign off (e.g. the workflow takes place over multiple days), then instance name can be used for that purpose. For example, a prefix such as [DAY 2 NEEDED] or Supervisor-ready or even can be added based on form answers.
If a supervisor finds the need for more work to be done, they don't sign off, so the form still can't be finalized. Again, the instance name can be used to indicate that a revisit is needed.
Once a supervisor signs off, the intent is for that submission to be sent without any further edits made. At that point they finalize and either autosend runs or the data collector sends when they are ready.
It sounds like you have a different workflow in mind. Can you please give us all the steps and exactly where having a truly finalized state is problematic? If you can share an example form that encodes your workflow, that would help make things more concrete.
You mention wanting to know whether draft forms have errors and wanting to jump to the first missing question which makes a lot of sense. First, note that if you have required supervisory questions that the data collector knows are blank, they can continue to use the "finalize" button for this purpose. We've also added in-form validation from ⋮ > Check for errors. This will do what you describe: it will jump to the first error found.
In the next release, we will add feedback about whether a form has errors even when saving as draft and we will also indicate whether or not a draft has any errors in the drafts list (see the "Saved with errors" prefix on the rightmost mockup in my post above).
Sorry, I don't like the idea of "overloading" and dynamically changing now the instanceName to mix-in a status/workflow concept.
We would prefer a status attribute/variable (maybe a metadata via setting sheet) which allows to clearly separate forms, even in different UI "folders"/menu items", still in progress at enumerator level (e.g. take a lunch break or meet household again next days) and forms declared as "finalised" by the enumerator, i.e. ready for local QA by/with supervisor.
We would also like that new concepts are similar in Collect and Enketo, very often we use both in the same survey. E.g. Enketo/KoboToolbox has a "validation_status" and a "Validate" option in preview.
Can you say more about why? This has always been one of the intended uses of instance name. Its advantage is that it's completely workflow-agnostic and allows you to decide exactly how you want the status information conveyed.
My understanding is that our recent and ongoing changes around finalization are neutral with respect to this unless your form design specifically allowed enumerators to finalize without supervisor approval and then finalize again with supervisor approval so that "finalized" had two different meanings. This is certainly possible but I would expect it to be quite rare. If this is what you did, it would be helpful to see your form design.
You are asking specifically for more built-in support for review workflows. We have considered this at various points over time but it feels like a separate endeavor that we likely won't be able to prioritize in the short term. We'll be sure to reach out as part of user research when we do get to it.
Did you know that the ODK team now maintains Enketo as well? We've been doing our best over time to harmonize concepts between it and Collect as we can. Both have a lot of history and large user bases so we do this iteratively and make sure it's guided by user research. Indeed, harmonization with Enketo is yet another reason we want to move in the direction of having a clear separation between drafts and finalized forms. In Enketo's offline mode, forms can either be saved as draft, in which case they are editable, or can be queued for sending, in which case they are not editable. We are working towards Collect matching that.
As you note, the preview mode of Enketo has a "Validate" button. What we typically hear from users is that it's not immediately obvious that this will check the data entered. Because it's only available in preview, it feels like it should be about the form definition. This is something we do hope to improve. As far as I know, the term "validate" and that functionality are not available anywhere else from Enketo but do let me know if I'm missing something. For Collect, we considered "Validate" at lengths when naming the new "Check for errors" functionality but again, it didn't feel clear enough what it was validating and what a valid form would mean (is it about the structure? Does it include required questions being filled?).
I am slightly uncomfortable with the process of Bulk Finalisation that you have described and it feels a bit like a retrograde step in that I think it loses the 'granularity' of being able to hold different forms at draft or final state. As @BartRoelandt2 indicated the current workflow used when sending multiple forms would intuitively make sense to me too - especially when you have the 'select all' option.
My experience is that 'tap and hold' to initiate multiple select or a 'select all' button is quite common and is a reasonable expectation for enumerators, and having the opportunity to finalise some forms and not others would appear to be more useful than all or nothing. My original suggestion of bulk finalise was implicitly thinking of the same process as for sending forms so that it didn't introduce visual clutter or complexity (naively assuming that you could replicate the process rather than start from scratch!).
Perhaps there are plans to remove that from the workflow of sending finalised forms?
One reason that I am concerned is that your suggestions for validation mean that I need to redesign existing forms to add that validation, and I had moved away from 'required' because enumerators can't move past the question unless it is filled in (unless that behaviour has changed?)...
Apologies if this appears negative, I understand how far backwards you are bending to be helpful and responsive
Thanks for the details, @seewhy. Do you think bulk finalized as described adds any value to you? Note that you could add just a single required question at the end to prevent finalization as appropriate. We could also consider dropping this work for now and continuing to gather feedback.
The major difference is that in “ready to send” the primary goal of a data collector is to send forms. In drafts, we don’t want to steer users towards workflows that involve bulk finalization and so we do want it to be something that end users have to know about to use rather than a primary action.
Not currently but we generally recommend using auto-send and not seeing that screen at all.
Can you actually add a little bit more detail around this, please? Here's what @BartRoelandt2 said earlier which matches my understanding of a case where someone wouldn't want to finalize everything:
Our sense for something like that is that because the numbers are few, you could do two passes: first identify the filled forms that need review and mark them in some way (e.g. check a box for a last question that says "needs further action" or maybe do nothing if a critical piece of information like species is missing), then bulk finalize. I realize it may not be your absolute ideal but it feels like a good balance between efficiency for your workflow and not interfering with other workflows we know are more common.
My understanding of your workflows was that you were mostly offline for a long time and then heading back to an area with connectivity for periodic submission. In that case, it seems everything could be a draft always up until the moment you're ready to send and so bulk finalization would be a good fit. Can you share some specific cases in which you would want to finalize a subset of filled forms, please?
The granularity I mean is about being able to make decisions about my data and when I think it is ready for submission. This has been a strength of ODK Collect that I have always been able to manage the data while it is on my device and I suppose I may even have developed 'bad' habits as a result. If I am using more than one form-set (I'm trying to differentiate between forms / surveys and records here) then they are not always ready to finalise at the same time - in my case it would be 1 or possibly 2 of Form A (the site description) and usually between 20 and 100 of Form B (detailed records for each location), but could be more. This actually arises from having moved away from deep nesting / repeats to simplify data collection on a 'point-by-point' basis (i.e. enumerators not having to navigate the 'add new...' and getting lost in the form). If I am at the same site for more than one day I would usually try to submit multiples of Form B (to avoid potential data loss) but not Form A just in case I need to change anything on that. I would need to make any edits to Form B on the same day either in the field or at the end of that session.
But with more discipline maybe I can work round it so maybe I'm being over-cautious and want too much control. This one feels like a 'breaking-change' so I have to adapt - it's not necessarily a bad one. Maybe I need to change Form A to introduce a 'required' question to prevent finalisation and not touch the design of Form B... I had nearly made Form A redundant by a conditional section that only shows for the first record, dependent on the site name from the previous record being the same but haven't amalgamated them. Maybe that's the cause of reluctance from those of us involved in this discussion - things were 'working' (or at least we'd got accustomed to them) and now they won't without our own intervention.
I recognise that for larger scale data collection there may be good reasons to discourage or even prevent editing a form so maybe, as you say, these requests don't meet those needs of the majority.
I still think bulk finalise is a good idea, so thanks for taking that on board. And hopefully we are more sure of the implications (I am!) of change.
Maybe you need a new job title: The Extraordinary Cat Herder
They're not bad habits, you used some things the tool could do, and now the tool is changing on you! I've been on the other side of that and I know how frustrating it is, even knowing that there's a bigger picture out there. I want to be clear that the whole team feels empathy for those having to rethink some of their processes because of what we're doing. We're really trying to make things as smooth for as many people as possible without ending up with the Homer car as @Aly_Blenkin likes to say.
This is a change I definitely support and that I hope has been overall helpful.
This sounds like it would be a good change. The question could even be something like "Have you sent all Form B records?" to force a sequence between the two. Even if everything stayed the same in Collect, that would prevent you from accidentally autosending the wrong forms if you were going too quickly, for example. In general, if there's a verification step that's required, I think it's very helpful to encode that in the form with a required question.
Hello @LN,
Thanks for explaining more.
Yes, we know and are glad you took over Enketo management. So, hopefully, differences will diminue (like regex and date behaviour
We would then have to program these changes of instanceName:
Saved Prefix_ (in work)
Ready for QA Prefix_ (i.e. "finalised" from enumerator view)
a) QA sucessful Prefix_ (might be with adfitional remark or
b) QA rejected Prefix_
...
Ready for Send Prefix_ (really "finalised", incl. QA)
Sent Done, only visible in menu (cannot be programmed in instanceName).
Until now we used "finalise" by enumerator to check for errors and then to transfer to QA by local supervisor.
A new check for error option (in the sandwich menu) would be great and can replace parts of our finalise usage.
Auto-send should not be the only option, please, esp. in remote areas. Sometimes we even use enumerator devices without Wifi/Internet and transfer via Blutooth/Hotspot (e.g. Supervisor or IT support in the field).
I know what you mean and agree those differences are very annoying. They're not at the very top of our current list (and they're unfortunately not as straightforward as they may seem) but we're aware of them.
What do you think of only adding a prefix for "ready for QA"? Here are how I think the other states can be identified:
in progress: no prefix, only available from drafts
ready for QA: this is where a prefix feels very helpful to me
QA successful: the reviewer can finalize or send depending on connectivity
Ready for send: this feels the same as qa successful to me so I'd need more info to differentiate the two
QA rejected: no prefix, only available from drafts; this feels generally similar to in progress. Alternately, you could include prefixes for common tasks that need to be done or even the full QA comment.
Here is a sample form with the general idea I have in mind. This would work the same way before and after finalization is made truly final.
How are you differentiating between in progress, ready for QA and QA successful currently? I see two options without a prefix:
you don't have a required question for QA. This means that in a world where finalization isn't final, you can finalize to indicate "ready for QA". It also means a "ready for QA" submission can accidentally be sent if there's an internet connection. Or if there's no internet connection, there's no way to differentiate between pre and post QA.
you do have a required question for QA. This means you can't finalize to indicate "ready for QA" and so you need to rely on memory to differentiate between in progress drafts and ready for QA.
It's available now in v2023.2!
We do recognize the value of manual send for certain workflows and don't have any plans to remove it.
can be added based on form answers.