Arabic values from itemsets show as question marks

cocco_rgnt · February 27, 2019, 11:41am

What is the problem? Please be detailed.
Hi everyone!
I'm working on some multi-language forms (English and Arabic) and I have an itemsets to use cascading selection for administration levels (Governorate/District/Sub-district/Village). Now, the questions shows properly in both English and Arabic, but when switching to Arabic the labels of the administration levels will show as a series of question marks.
The labels show properly in Arabic when I open the csv file, so I don't know if the problem is just that ODK doesn't support Arabic from external itemsets?

What ODK tool and version are you using? And on what device and operating system version?
I'm using ODK Collect v1.19 on a Lenovo Tab4 8 Plus with Android 8.1

What steps can we take to reproduce the problem?
Please see attached test form + itemsets file.

What have you tried to fix the problem?
I tried to use Italian as a second language and that worked fine, so I guess it's a problem of decoding Arabic letters.

Anything else we should know or have? If you have a test form or screenshots or logs, attach here.
Please see attached files.ArabicTest.xls (28.5 KB)
itemsets.csv (187.8 KB)

Thank you very much to anyone who will reply.

Francesco

Grzesiek2010 · February 27, 2019, 4:32pm

Hi @cocco_rgnt

Everything is fine o the Collect's side. The problem is with your itemsets.csv file

sample:

distrlist,SY0200,Jebel Saman,ÌÈá ÓãÚÇä,SY02,,
distrlist,SY0202,Al Bab,ÇáÈÇÈ,SY02,,
distrlist,SY0203,Afrin,ÚÝÑíä,SY02,,
distrlist,SY0204,A'zaz,ÇÚÒÇÒ,SY02,,
distrlist,SY0205,Menbij,ãäÈÌ,SY02,,
distrlist,SY0206,Ain Al Arab,Úíä ÇáÚÑÈ,SY02,,
distrlist,SY0207,As-Safira,ÇáÓÝíÑÉ,SY02,,
distrlist,SY0208,Jarablus,ÌÑÇÈáÓ,SY02,,
distrlist,SY0700,Idleb,ãÑßÒ ÅÏáÈ,SY07,,
distrlist,SY0702,Al Ma'ra,ãÚÑÉ ÇáäÚãÇä,SY07,,
distrlist,SY0703,Harim,ÍÇÑã,SY07,,
distrlist,SY0704,Jisr-Ash-Shugur,ÌÓÑ ÇáÔÛæÑ,SY07,,
distrlist,SY0705,Ariha,ÃÑíÍÇ,SY07,,
distrlist,SY0800,Al-Hasakeh,ãÑßÒ ÇáÍÓßÉ,SY08,,
distrlist,SY0802,Quamishli,ÇáÞÇãÔáí,SY08,,
distrlist,SY0803,Al-Malikeyyeh,ÇáãÇáßíÉ,SY08,,
distrlist,SY0804,Ras Al Ain,ÑÃÓ ÇáÚíä,SY08,,
distrlist,SY0901,Deir-ez-Zor,ãÑßÒ ÏíÑ ÇáÒæÑ,SY09,,
distrlist,SY0902,Abu Kamal,ÇáÈæßãÇá,SY09,,
distrlist,SY0903,Al Mayadin,ÇáãíÇÏíä,SY09,,
distrlist,SY1101,Ar-Raqqa,ãÑßÒ ÇáÑÞÉ,SY11,,
distrlist,SY1102,Tell Abiad,Êá ÃÈíÖ,SY11,,
distrlist,SY1103,Ath-Thawrah,ÇáËæÑÉ,SY11,,

so your arabic translations are broken. If you fix it everything should be fine.

cocco_rgnt · February 27, 2019, 4:38pm

Hi,
Thanks for your reply!
My itemsets.csv seems to work fine for me though and still Arabic labels are not showing properly.
Am I doing something wrong there?

list_name	name	label::English (en)	label::Arabic (ar)
govlist2	SY02	Aleppo	حلب
govlist2	SY07	Idleb	إدلب
govlist2	SY08	Al-Hasakeh	الحسكة
govlist2	SY09	Deir-ez-Zor	دير الزور
govlist2	SY11	Ar-Raqqa	الرقة

distrlist	SY0200	Jebel Saman	جبل سمعان
distrlist	SY0202	Al Bab	الباب
distrlist	SY0203	Afrin	عفرين
distrlist	SY0204	A'zaz	اعزاز
distrlist	SY0205	Menbij	منبج
distrlist	SY0206	Ain Al Arab	عين العرب
distrlist	SY0207	As-Safira	السفيرة
distrlist	SY0208	Jarablus	جرابلس
distrlist	SY0700	Idleb	مركز إدلب
distrlist	SY0702	Al Ma'ra	معرة النعمان
distrlist	SY0703	Harim	حارم
distrlist	SY0704	Jisr-Ash-Shugur	جسر الشغور
distrlist	SY0705	Ariha	أريحا
distrlist	SY0800	Al-Hasakeh	مركز الحسكة
distrlist	SY0802	Quamishli	القامشلي
distrlist	SY0803	Al-Malikeyyeh	المالكية
distrlist	SY0804	Ras Al Ain	رأس العين
distrlist	SY0901	Deir-ez-Zor	مركز دير الزور
distrlist	SY0902	Abu Kamal	البوكمال
distrlist	SY0903	Al Mayadin	الميادين
distrlist	SY1101	Ar-Raqqa	مركز الرقة
distrlist	SY1102	Tell Abiad	تل أبيض
distrlist	SY1103	Ath-Thawrah	الثورة

Cheers,

Francesco

Grzesiek2010 · February 27, 2019, 4:43pm

If I open the file you attached it looks like:

So that's the problem. Do you see all of them in arabic properly if you open the file?

cocco_rgnt · February 27, 2019, 4:50pm

Yes they're all in Arabic when I open the csv:

On my laptop the region settings are set on Arabic:

If I set it on English, when I open the csv I see it as in the screenshot you shared, do you have any advise on how to fix that?

Grzesiek2010 · February 27, 2019, 5:56pm

The file should be saved using UTF-8 encoding. Look I just opened your file using Arabic character set copied the entire column to another file and saved using UTF-8, result:
itemsets.csv (210.9 KB)
seems fine.

cocco_rgnt · February 28, 2019, 8:16am

That worked, thanks mate!