As we have quite a dynamic workflow, converting external csv on a regular base to remove the straight apostrops in a specific "name" column is not feasable evenmore if alternatively we use entities instead of external csv (although in this case I think it also works in Enketo).
Has anybody experience with this issue and maybe a solution?
Many of the earlier discussions were related with proper UTF-8 incolding when exporting csv from Excel. I haven't found a discussion directly related to straight apostrophs.
But the problem persits even if the csv is exported from Visual Code as pure UTF-8. The "mistake" only occurs with straight appostophs ('), all other special characters liike é, è, ä, ö, ô work perfectly well.
My guess is, that Enketo interpretes the straight apostroph some sort as code and then fails on execution.
I saw that a new Web Form is in development that will replace Enketo. That's good news. Do we know for when the release is planned?
Could you replace pulldata with an instance lookup and see if it behaves?
(only if you are displaying / populating text fields) when you pull the values, could you wrap the lookup or result in a translate to replace the ' with another apostrophe type character? I'm not actually sure this is possible with pulldata.
Indeed the behaviour disappears when using instance instead of pulldata, even without using translate().
I had some issues with instance where pulldata worked better. I will do some further testing. There are still some mysteries to me around how pulldata and instance differ.
Prepopulating select_from_file questions from an entity or an external csv with values containing straight apostrophs (') also work in Enketo if instance is used instead of pulldata.
So sorry, this is probably the messiest area of ODK. The instance syntax has been part of the form specification since its very beginnings. It's really general -- you can use it to get values from any kind of list of values that can be represented: choices from the choice list, attached CSVs, attached geoJSON, etc. You can also filter the list in many different ways.
pulldata was introduced to Collect as a way to get better performance for the common case of getting a single value out of a CSV by doing an exact match between two values. It can be used in place of instance expressions that have an expression in square brackets like "column = ${some_form_value}". In Collect, it uses a database representation of the CSV data and so can be used with larger CSVs.
In Enketo, pulldata was introduced as a shortcut to the corresponding instance expression. Because pulldata is more constrained than instance, Enketo uses the browser's built-in, optimized evaluator directly to evaluate it. On the other hand, instance expressions not from pulldata use a wrapper on that evaluator. The only thing I can think of to explain the behavior you're experiencing at the moment is that apostrophes are causing a problem for the browser evaluator.
It has been our dream for a long time to truly make pulldata truly identical to its corresponding instance expression. This will be the case in web forms and we're getting close to being able to do that in Collect as well.
The only known difference is performance. Anything else you should definitely let us know about.
I replaced pulldata with instance in the more complex forms. The problem in Enketo reliably no longer occurs with instance.
With some more testing and trying and headaches finally all forms perform well also with instance. The issues with instance were related to pulling data into select lists. Somehow (sorry for not being more precise) the issues were resolved by using a trigger which was not needed with pulldata. But I didn't look any deeper. Great it works now.
