We keep getting errors on our new Samsung S9s using Android 8.0.0 and ODK Collect 1.50.0 and up when using a pre-loaded CSV file to populate a Select question. The form will function perfectly for awhile then pop up with the error seen in the attached file. Sometimes deleting the blank form will solve the problem, other times I've had to completely wipe ODK Collect from the phone and reload everything. On our older phones using Android 6 or 7, we've never had this issue using the same form and versions of ODK. Any ideas of how to fix, or where to submit a bug report? If you need any other info, please let me know.
This is as good as a place as any to report a bug report. @Grzesiek2010 and @LN any ideas what this could be?
1 Like
could you attach your form? I tested pre-load.zip (9.0 KB)
and it looks fine.
2 Likes
This has the XML and CSV we are using.
LindgrenQuickCheck.zip (2.3 KB)
@Greg_Pommen are you sure you put your csv file in /odk/forms/LindgrenQuickCheck-media/ dir not just in /odk/forms/ where the xml file is?
2 Likes
Given that this works some times but not others it's unlikely @Greg_Pommen is doing anything wrong.
This error message means that the exists() call on the .db file created from the csv is returning false. Could something about permissions or data access have changed in Android 8? I've been looking for ways that exists() could fail when the file is there but am not having much luck.
2 Likes
I tried Android 8 + Collect v1.15.0 + attached forms and can't reproduce.
2 Likes
@Grzesiek2010, I'm just using the standard forms Xforms upload in the ODK AppEngine. It doesn't seem to give me any issues on any other phones, just the ones I stated in the post. @Grzesiek2010, it doesn't happen right away usually, it seems to work, often for days, then just stops and give the error. One of our workers, just texted me that she's getting the same error again, on a completely different form that uses CSV's as well, that has been working, even on her phone just fine for many days.
I'm sorry I can't get this error to be a little more repeatable. I wish I could be more help.
I'm hoping this doesn't sound snarky, but we've been using ODK on our phones for data collection for about 6 years now. So we are reasonably familiar with how the system works. This is a completely new error, and only with the ODK Collect, Android, and Samsung versions of the phone I mentioned. Otherwise, ODK in our environment is pretty much 99.9999999% reliable.
2 Likes
Doesn't come across snarky at all, @Greg_Pommen. We just need to work together to find some way to reproduce it. The S9 was just released in March and my guess is that there are bugs in the custom firmware they layer on top of Android.
@Grzesiek2010 Do you have any S9s accessible to test?
@Greg_Pommen Assuming we have an S9, is there are a set of steps you can take from a reboot that would cause his problem to happen every time?
2 Likes
No, I don't.
I investigate how pre-loaded CSV works:
- At first, it's a form and a CSV file in a media dir:
/odk/forms/LindgrenQuickCheck.xml
/odk/forms/LindgrenQuickCheck-media
/odk/forms/LindgrenQuickCheck-media/TrapLists.csv - Starting the form for the first time a database file is generated:
/odk/forms/LindgrenQuickCheck-media/TrapLists.db
and the CSV file is renamed to
/odk/forms/LindgrenQuickCheck-media/TrapLists.csv.imported
Info provided by Greg indicates those files are removed (or at least the /odk/forms/LindgrenQuickCheck-media/TrapLists.db file is removed).
@Greg_Pommen
Could you confirm that a corresponding media dir is empty when that error occurs? Or as I mentioned there is lack of /odk/forms/LindgrenQuickCheck-media/TrapLists.db file?
Is it possible you have installed any fishy software on that device? Maybe not really reliable antivirus which removes files from your device or something like that?
5 Likes
Hi, just a quick update until I can get the error to reoccur.
As far as I can tell there is no user installed weird software beyond an app called Light Meter (seems harmless enough), and a corporate required app called UEM Client (made by Blackberry). We have the same apps on other phones as well that aren't having this issue. Samsung has plenty of its own weird software. I could probably find a way to generate some sort of hardware/software profiles if that would help.
There are 3 files in the /odk/forms/Lindgren Quick Check-media folder:
/odk/forms/Lindgren Quick Check-media/TrapLists.db
/odk/forms/Lindgren Quick Check-media/TrapLists.csv.imported
/odk/forms/Lindgren Quick Check-media/TrapLists.db-journal
Other forms that use CSV files have the same .db, .csv.imported, .db-journal extensions. I doubled checked an older Samsung Note 4 running Android 6.0.1 and ODK Collect v1.15.1 and it's media files are the same.
I'll see what changes when I finally reproduce the error.
1 Like
@Grzesiek2010, @LN, @yanokwa, I was finally able to reproduce the error, but all the corresponding media files with .db, .db-journal, and .csv.imported are still in their respective form media folders. I checked the permissions and they read as -RW on all of those files as do the corresponding .xml files. I'm stumped!
Could you copy at least that media dir and attach here when you see that error. I'm wondering if it works on any other device (I bet it doe's taking into account what you said before).
2 Likes
We can only guess at this point but my guess would be that it's something in the state of the device or some bug in the exists code that is causing the issue.
But perhaps @Grzesiek2010 is right that there's something strangely corrupt about the files. @Greg_Pommen if you're comfortable sharing your media dir, you can zip it up and either attach it or for more privacy, private message it to @Grzesiek2010 for further investigation.
1 Like
There's nothing especially private about this info, so I'm uploading a zipped copy of the directory.Lindgren Quick Check-media.zip (2.0 KB)
Let me know if there's anything else you need. Thank you!
1 Like
It works for me... what a surprise. Hmm I investigated our reports and you are not alone.
Reports from v1.15.0 and v1.15.1:
- over 400 occurrences
- 30 affected users
- devices like Samsung, Crosscall, Tecno
- Android 6, 5, 7, 4, 8
Maybe some of them just forgot to add media files (I hope) but still.
@Greg_Pommen you said it works then it stops... what do you have to do to make the form working again? Just open the form again? Or maybe even if you open the form a few times the problem still occurs?
3 Likes
@Grzesiek2010 I'm somewhat glad we aren't alone, but not really since there is still an issue.
Sometimes deleting the form with the CSV, and then reloading it will solve the issue, for awhile. Two days ago, one user told me she was having to delete and reload the form every time she needed to submit a form. Other times I've had to completely wipe ODK Collect from the phone, the odk folder too, and reinstall ODK, plus the forms.
Let me know what else I can do to assist in solving this issue. Normally ODK is virtually 100% reliable, that's why this is so surprising.
30 users out of hundreds of thousands is pretty small, but still!
@Grzesiek2010 Do you know if this is a regression?
.exists only guarantees that if it returns true the file exists and is readable: https://developer.android.com/reference/java/io/File.html#exists(). It could return some false negatives.
The best short-term experiment I can think of is to remove the call on exists(). This will mean that if the file really doesn't exist, users will get an error that looks like no such table: externalData (code 1):, while compiling: SELECT c_name_key, c_name FROM externalData. It's not pretty but it will give us some more information if it does keep happening. Let's continue the code discussion at https://github.com/opendatakit/collect/issues/2357