Cannot download external instances in ODK Collect, from a custom implementation of Open Rosa

mbaudier · October 13, 2021, 10:20am

Hello,

I have developed a custom implementation of the Open Rosa protocol so that ODK Collect can "talk" directly with our (Java) system. It went very smoothly and it is working fine, thanks to the excellent documentation!

However, I cannot get external instances to work. I have implemented the whole chain (manifest referenced in the form list, and then download URL in the manifest itself), but ODK Collect refuses to update the form, without an error message. Android sometimes says that it has crashed, and I have sent an error report / feedback this morning via Android, so maybe you have received it.

I can see on the server that the web services are properly called, in the order one would expect:

217.9.50.47 - - [13/Oct/2021:10:05:42 +0000] "GET /api/odk/formList HTTP/1.1" 401 - "-" "org.odk.collect.android/v2021.2.4 Dalvik/2.1.0 (Linux; U; Android 9; SM-G390F Build/PPR1.180610.011)"
217.9.50.47 - - [13/Oct/2021:10:05:42 +0000] "GET /api/odk/formList HTTP/1.1" 200 767 "-" "org.odk.collect.android/v2021.2.4 Dalvik/2.1.0 (Linux; U; Android 9; SM-G390F Build/PPR1.180610.011)"
217.9.50.47 - - [13/Oct/2021:10:05:42 +0000] "GET /api/odk/formManifest/example/form/SimpleExampleForm/manifest HTTP/1.1" 200 359 "-" "org.odk.collect.android/v2021.2.4 Dalvik/2.1.0 (Linux; U; Android 9; SM-G390F Build/PPR1.180610.011)"
217.9.50.47 - - [13/Oct/2021:10:05:47 +0000] "GET /api/odk/form/example/form/SimpleExampleForm HTTP/1.1" 200 13253 "-" "org.odk.collect.android/v2021.2.4 Dalvik/2.1.0 (Linux; U; Android 9; SM-G390F Build/PPR1.180610.011)"
217.9.50.47 - - [13/Oct/2021:10:05:47 +0000] "GET /api/odk/formManifest/example/form/SimpleExampleForm/manifest/tag HTTP/1.1" 200 110 "-" "org.odk.collect.android/v2021.2.4 Dalvik/2.1.0 (Linux; U; Android 9; SM-G390F Build/PPR1.180610.011)"

In the form the secondary instance looks like:

...
<instance id="tag" src="jr://file/example/terms/tag.xml"/>
...

The manifest is:

<?xml version='1.0' encoding='UTF-8' ?>
<manifest xmlns="http://openrosa.org/xforms/xformsManifest">
<mediaFile>
<filename>example/terms/tag.xml</filename>
<hash>md5sum:b984d68cd0bee21384ca51cef7c90880</hash>
<downloadUrl>https://examples-unstable.work.argeo.pro/api/odk/formManifest/example/form/SimpleExampleForm/manifest/tag</downloadUrl>
</mediaFile>
</manifest>

And the download URL returns:

<?xml version="1.0" encoding="UTF-8"?><tag><term name="important"/><term name="long"/><term name="new"/></tag>

I have tried to pull the form using ODK BriefCase, and it seems happy with it:

2021-10-13 11:03:05,522 [ForkJoinPool-4-worker-3] INFO o.o.b.p.a.PullFromAggregateTracker - Pull SimpleExampleForm - Start pulling form and submissions
2021-10-13 11:03:05,536 [ForkJoinPool-4-worker-3] INFO o.o.b.p.a.PullFromAggregateTracker - Pull SimpleExampleForm - Start downloading form
2021-10-13 11:03:05,692 [ForkJoinPool-4-worker-3] INFO o.o.b.p.a.PullFromAggregateTracker - Pull SimpleExampleForm - Form downloaded
2021-10-13 11:03:05,699 [ForkJoinPool-4-worker-3] INFO o.o.b.p.a.PullFromAggregateTracker - Pull SimpleExampleForm - Start getting form manifest
2021-10-13 11:03:05,737 [ForkJoinPool-4-worker-3] INFO o.o.b.p.a.PullFromAggregateTracker - Pull SimpleExampleForm - Got the form manifest
2021-10-13 11:03:05,741 [ForkJoinPool-4-worker-3] INFO o.o.b.p.a.PullFromAggregateTracker - Pull SimpleExampleForm - Start downloading form attachment 1 of 1
2021-10-13 11:03:05,769 [ForkJoinPool-4-worker-3] INFO o.o.b.p.a.PullFromAggregateTracker - Pull SimpleExampleForm - Form attachment 1 of 1 downloaded
...
2021-10-13 11:03:06,346 [AWT-EventQueue-0] INFO XFormParser - Reading XML and parsing with kXML2 finished in 3.021 ms
2021-10-13 11:03:06,349 [AWT-EventQueue-0] INFO XFormParser - Consolidating text finished in 1.830 ms
2021-10-13 11:03:06,350 [AWT-EventQueue-0] INFO XFormParser - Parsing form...
2021-10-13 11:03:06,350 [AWT-EventQueue-0] INFO XFormParser - Title: "Example Session"
...
2021-10-13 11:03:06,406 [AWT-EventQueue-0] INFO XFormParser - Creating FormDef from parsed XML finished in 56.322 ms

So, the problem only appears on ODK Collect. I must be doing something wrong, but I cannot find logs on the mobile phone.

How do you think I should proceed from here?
Thanks in advance!

Mathieu

PS: This testing environment is only with dummy data, so I can easily provide an access if someone wants to reproduce directly.

mbaudier · October 17, 2021, 8:21am

Hello,

I have debugged using Android Studio and the problem seems to be that our paths to the external secondary instance have subdirectories:
<instance id="tag" src="jr://file/example/terms/tag.xml"/>

<filename>example/terms/tag.xml</filename>

Which leads to this crash:

2021-10-17 08:28:50.858 9814-9937/? E/AndroidRuntime: FATAL EXCEPTION: AsyncTask #3
Process: org.odk.collect.android, PID: 9814
java.lang.RuntimeException: An error occurred while executing doInBackground()
at android.os.AsyncTask$3.done(AsyncTask.java:354)
at java.util.concurrent.FutureTask.finishCompletion(FutureTask.java:383)
at java.util.concurrent.FutureTask.setException(FutureTask.java:252)
at java.util.concurrent.FutureTask.run(FutureTask.java:271)
at android.os.AsyncTask$SerialExecutor$1.run(AsyncTask.java:245)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1167)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:641)
at java.lang.Thread.run(Thread.java:764)
Caused by: java.lang.RuntimeException: Could not copy '/storage/emulated/0/Android/data/org.odk.collect.android/files/projects/ac2d95cd-c319-4f58-9a90-3a2f91ccaefc/.cache/download-b7854eef-0a51-4477-9172-47a2677450ac/tag.xml3517133847896120566.tempDownload' over '/storage/emulated/0/Android/data/org.odk.collect.android/files/projects/ac2d95cd-c319-4f58-9a90-3a2f91ccaefc/.cache/download-b7854eef-0a51-4477-9172-47a2677450ac/media/example/terms/tag.xml'. Reason: /storage/emulated/0/Android/data/org.odk.collect.android/files/projects/ac2d95cd-c319-4f58-9a90-3a2f91ccaefc/.cache/download-b7854eef-0a51-4477-9172-47a2677450ac/media/example/terms/tag.xml (No such file or directory)
at org.odk.collect.android.formmanagement.ServerFormDownloader.writeFile(SourceFile:368)
at org.odk.collect.android.formmanagement.ServerFormDownloader.downloadMediaFiles(SourceFile:389)
at org.odk.collect.android.formmanagement.ServerFormDownloader.processOneForm(SourceFile:109)
at org.odk.collect.android.formmanagement.ServerFormDownloader.downloadForm(SourceFile:81)
at org.odk.collect.android.tasks.DownloadFormsTask.doInBackground(SourceFile:62)
at org.odk.collect.android.tasks.DownloadFormsTask.doInBackground(SourceFile:41)
at android.os.AsyncTask$2.call(AsyncTask.java:333)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at android.os.AsyncTask$SerialExecutor$1.run(AsyncTask.java:245)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1167)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:641)
at java.lang.Thread.run(Thread.java:764)

Looking at the code in the v2021.2.x of ODK Collect:

src/main/java/org/odk/collect/android/formmanagement/ServerFormDownloader.java
Line 360
String errorMessage = FileUtils.copyFile(tempFile, destinationFile);

but the parent directories of the destination file are not created before, and not in FileUtils:

src/main/java/org/odk/collect/android/utilities/FileUtils.java
line 198
private static String actualCopy(File sourceFile, File destFile) {

I hacked our code so that it uses file paths without hierarchy (that is tag.xml instead of example/terms/tag.xml), and it worked!

I had a quick look at the master branch, and while ServerFormDownloader has changed, it still calls the same methods in FileUtils, which haven't changed.

My understanding of the Open Rosa specifications is that one can use hierarchical path structures:
https://docs.getodk.org/openrosa-form-list/#the-manifest-document
<filename>path/to/agilefrog.png</filename>

Are you interested that I test further on the master branch, and try to provide a patch?
Or do you prefer to look at it directly?

Cheers,

Mathieu

mbaudier · November 3, 2021, 4:45am

Hello,

Should I rather post this issue to the Support forum?
Or as a GitHub issue? (https://github.com/getodk/collect/issues)

Cheers,

Mathieu

seadowg · November 3, 2021, 9:27am

Hey Mathieu ! There's definitely something wrong here as Collect ideally shouldn't crash in this scenario.

Two questions:

Why do the media files (in this case an external instance) have to have hierarchical path structures in this case? I'm thinking that the example in the spec might be a mistake as the node name (filename) and how Collect treats it suggests it's meant to be a literal "file name" rather than a "file path". I'm not sure about the history of that however!
Could you try to see if your server works with an older version of Collect? Releases are here.