CSV column heading " error

Screenshot from 2023-05-10 10-10-25
pizza_questionnaire2.xlsx (8.1 KB)
topping.csv (114 Bytes)

Hi all, I'm trying to test a simple form (as attached) but prompted 'CSV column heading " ' error. It uses select_one_from_file feature. Fyi, the Central server is locally hosted. Your guidance is very much appreciated.

Thanks,
Kelv

What version of Central are you running?
I checked in 2032.2 and it loaded fine in Enketo

Screenshot from 2023-05-10 14-23-26

Hi Andrew. My Central version is as attached.

Could this be Ubuntu docker user permission on folder issue?

Tq.

I published the form without test. Downloaded, filled and sent from ODK Collect. It works like what were said in discussion below.

Enketo preview with external file doesn’t work

Thanks,
Kelv

I'm glad to hear it's working in Collect but very strange that the web form isn't working for you, I would expect it to and @ahblake's has confirmed it can work. The thread you link to is from 2017 and web forms now work with CSV files.

The error message you got is strangely cut off, the full message is CSV column heading "${header}" cannot be turned into a valid XML element. For some reason Enketo appears to try to do something with a blank column heading.

We'll try to investigate more soon but will consider it low priority unless others run into it. Thanks for updating the thread with your latest status!

1 Like

Hi, I am also having the same error in a form that is already in use since months. For some reason, the form Project Manager and the Administrator of ODK Central cannot edit submissions anymore as this message appears. Any suggestion?

I have been investigating the issue reported by @neusrosell and finally have been able to fix them by two methods:

Either modifying the DNS records so even from inside the host machine, the resolved domain name is a public IP address and not the internal private address (and rebooting the machine)

Or either adding to files/enketo/config.json.template and service/config.json.template the section:

    "ip filtering": {
        "allowPrivateIPAddress": true,
        "allowMetaIPAddress": false,
        "allowIPAddressList": [],
        "denyIPAddressList": []
    },

The fixing consist on setting the value ture for "allowPrivateIPAddress"

After that, it is necessary to rebuild the containers for the change to take effect.
In the rebuild process, you can lose access to your database. Make a Backup first and follow this fix:

https://docs.getodk.org/central-troubleshooting/#database-reset-after-running-docker-command
docker

Actually, the root cause of the problem occurs inside ODK Central "sevice" and "enketo" docker containers:

When Enketo tries to fetch the CSV file, an error occurs and a result code of "500 Internal Server Error" is returned, but the request returns a body that is not a CSV but an HTML text describing the error:

<!DOCTYPE html><html lang="en-us" dir="ltr"><head><title>Enketo</title><meta charset="utf-8"><meta name="author" content="Martijn van de Rijdt (Enketo LLC)"><meta name="viewport" content="width=device-width, initial-scale=1.0"><link rel="shortcut icon" href="/-/images/favicon.ico"><link rel="icon" type="image/png" sizes="180x180" href="/-/images/icon_180x180.png"><link rel="apple-touch-icon" sizes="180x180" href="/-/images/icon_180x180.png"><link rel="stylesheet" href="/-/css/error.css"></head><body><h1 class="error">DNS lookup 10.20.30.50(family:4, host:odk.palestinercs.org) is not allowed. Because, It is private IP address.<span class="status-code">Error Code: 500</span></h1></body></html>

For clarity and without HTML code it says:

"DNS lookup 10.20.30.50(family:4, host:odk.palestinercs.org) is not allowed. Because, It is private IP address.Código de error: 500”

Javascript code of Enketo page running in the web browser does not detect the 500 HTML result code and continues to try to process the retuned body of the CSV-file request normally, so it interprets the HTML text as a CSV file and thinks that the name of the first column is up to the fist comma. I.E.:

<!DOCTYPE html><html lang="en-us" dir="ltr"><head><title>Enketo</title><meta charset="utf-8"><meta name="author" content="Martijn van de Rijdt (Enketo LLC)"><meta name="viewport" content="width=device-width

This text contains spaces and is thus an invalid CSV column name for Enketo and that is the reason of the appearing error.

An additional problem is that when this "column header" text is inserted in the error message:

CSV column heading "${header}" cannot be turned into a valid XML element

the contents of ${header} are not afterward properly HTML-escaped to be rendered on the web page showing the error, producing a resulting web page HTML code that is invalid and is cut in the middle of the message, although you can see the complete "CSV column" name if you go to the HLML source of the error page:

Enketo error in HTML.zip (54.8 KB)

Thanks for that thorough troubleshooting, @ehbossa, and for sharing your results.

Can you tell us a little bit about how you set up your server in the first place? Did you deviate significantly from https://docs.getodk.org/central-install-digital-ocean/? Did you change the Docker compose file significantly? Or do you have some other network layer sitting in front of your server?

The good news is that we will be removing the ip filtering section and functionality from the Enketo soon because the problem it was intended to address has been resolved in other ways. So this should stop occurring with the next major Enketo release.

Still, if we can figure out more about what's going on we might be able to address it sooner.

This should no longer happen in v2023.2 and later. Please make sure you're upgrading to get important security updates! Even previously, it only happened when running docker-compose down instead of docker-compose stop.

Hi @LN ,
I'm a colleague, sorry for not responding, I'm replying to the questions and another bug we've seen.

  • Can you tell us a little bit about how you set up your server in the first place?
    versions:
    24ee74e5f974a518aa1cc8b06e7addb3be6b4690 (v1.3.3-2-g24ee74e)
    5cc6fd79d112ce36d6298c61bb8817689c4c323b client (v1.3.2)
    1d1a3a59969e61383da74119e405e67778b7a170 server (v1.3.3)
    OS ubutnu
  • Did you deviate significantly from https://docs.getodk.org/central-install-digital-ocean/ 1?
    No the only modifcation was files/enketo/config.json
  • Did you change the Docker compose file significantly?
    No. Only add a new repository because the ubuntu is deprecated
  • Do you have some other network layer sitting in front of your server?
    No.

On the other hand, the new bug we've seen is that when a form uses a CSV file for a list of options, it gives the same error as before.
The encoding is UTF-8




I've lost track of the different servers involved here! @sowe1 is yours the same as @ehbossa's? And that's the one that worked after a configuration change?

This looks like the same issue to me and the same as Media file generate a bad path as well. You can see from the 500 error that Enketo is unable to request the CSV from Central because of a private IP address issue.

The possibility of this problem has been removed in recent Central versions. v1.3 is now getting pretty old and I would strongly recommend upgrading as soon as you can. You'll want to make sure to have a backup and plan for a maintenance window to go through the several changes that have happened between that version and current ones.

Hi @LN , @ehbossa and I are part of the same team and if we are talking about the same server. If I agree, but the solution is the same as what @ehbossa did, which is to activate the 'allowPrivateIPAddress' option setting the value to True. That's why I was surprised to see the error and hence if something needs to be modified in the enketo.

Yes, I agree that we need to update the ODK. We are figuring out how to do it without losing any information (which could be solved with a database backup) and without losing the versions of the forms or losing images, etc. That part isn't very clear to us yet.

Centra upgrades are typically very reliable and you definitely shouldn't lose any data. We recommend you do a full machine backup because that's the easiest way to rollback if something does go wrong.