Some filled forms and media not exporting in Briefcase

Hiya folks,

What is the problem? Please be detailed.
Two separate problems consolidated here:
[1] We are running surveys on Galaxy note tablets in South America doing household and market surveys. 10 of the Market surveys from one date will not export in Briefcase (we do it all manually and not via server given poor internet connection). Survey instances before and after work fine. The briefcase log is at the end of this message

[2] We are taking several photos of the market's inventories and are having the problem of only the first photo (of an optional 1-10) exporting per survey instance (per market). I read up on a similar problem elsewhere but there seemed to not have been a clear solution.. hopefully I'm wrong! The CSV output lists all media files and only exports the first photo taken per survey. We can unencrypt the survey and view them on the tablet that way but that's less desirable..

What ODK tool and version are you using? And on what device and operating system version?
We use Briefcase 1.11.2 and offline converter. The tablets are the latest galexy note tablets running ODK Collect.

Thanks for any help!!
Khristopher

Anything else we should know or have? If you have a test form or screenshots or logs, attach here.

2018-07-23 18:04:32,907 [Thread-38] ERROR o.o.briefcase.ui.export.ExportPanel - Error while exporting forms
org.opendatakit.briefcase.reused.BriefcaseException: org.xmlpull.v1.XmlPullParserException: unterminated entity ref (position:ENTITY_REF &l�@1:9 in java.io.InputStreamReader@2909bfb1) 
	at org.opendatakit.briefcase.export.SubmissionParser.parse(SubmissionParser.java:203)
	at org.opendatakit.briefcase.export.SubmissionParser.decrypt(SubmissionParser.java:171)
	at org.opendatakit.briefcase.export.SubmissionParser.lambda$parseSubmission$5(SubmissionParser.java:142)
	at java.util.Optional.flatMap(Optional.java:241)
	at org.opendatakit.briefcase.export.SubmissionParser.parseSubmission(SubmissionParser.java:122)
	at org.opendatakit.briefcase.export.ExportToCsv.lambda$export$3(ExportToCsv.java:109)
	at java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:193)
	at java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1382)
	at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:481)
	at java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:471)
	at java.util.stream.ForEachOps$ForEachOp.evaluateSequential(ForEachOps.java:151)
	at java.util.stream.ForEachOps$ForEachOp$OfRef.evaluateSequential(ForEachOps.java:174)
	at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234)
	at java.util.stream.ReferencePipeline.forEach(ReferencePipeline.java:418)
	at org.opendatakit.briefcase.export.ExportToCsv.export(ExportToCsv.java:112)
	at org.opendatakit.briefcase.ui.export.ExportPanel.lambda$export$10(ExportPanel.java:199)
	at java.util.stream.ForEachOps$ForEachOp$OfRef.accept(ForEachOps.java:184)
	at java.util.stream.ReferencePipeline$11$1.accept(ReferencePipeline.java:373)
	at java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1382)
	at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:481)
	at java.util.stream.ForEachOps$ForEachTask.compute(ForEachOps.java:291)
	at java.util.concurrent.CountedCompleter.exec(CountedCompleter.java:731)
	at java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:289)
	at java.util.concurrent.ForkJoinTask.doInvoke(ForkJoinTask.java:401)
	at java.util.concurrent.ForkJoinTask.invoke(ForkJoinTask.java:734)
	at java.util.stream.ForEachOps$ForEachOp.evaluateParallel(ForEachOps.java:160)
	at java.util.stream.ForEachOps$ForEachOp$OfRef.evaluateParallel(ForEachOps.java:174)
	at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:233)
	at java.util.stream.ReferencePipeline.forEach(ReferencePipeline.java:418)
	at org.opendatakit.briefcase.ui.export.ExportPanel.export(ExportPanel.java:188)
	at java.lang.Thread.run(Thread.java:748)
Caused by: org.xmlpull.v1.XmlPullParserException: unterminated entity ref (position:ENTITY_REF &l�@1:9 in java.io.InputStreamReader@2909bfb1) 
	at org.kxml2.io.KXmlParser.exception(Unknown Source)
	at org.kxml2.io.KXmlParser.error(Unknown Source)
	at org.kxml2.io.KXmlParser.pushEntity(Unknown Source)
	at org.kxml2.io.KXmlParser.nextImpl(Unknown Source)
	at org.kxml2.io.KXmlParser.nextToken(Unknown Source)
	at org.kxml2.kdom.Node.parse(Unknown Source)
	at org.kxml2.kdom.Document.parse(Unknown Source)
	at org.opendatakit.briefcase.export.SubmissionParser.parse(SubmissionParser.java:200)
	... 30 common frames omitted

Hi, @knicholas!

I'm sorry you're having issues with Briefcase.

It would help me a lot to have the blank form and a submission that's failing because we're already testing encrypted but we obviously left some scenario out.

Could you attach those here or send me an email with them to ggalmazor@gmail.com, please?

Email coming your way!

1 Like

Hi, @knicholas!

I just wanted to say that I've been out for 2 weeks but I'm now back and will work on this asap.

1 Like

Hi, I am having this same issue with the exact same specification (Briefcase 1.11.2). Only the first 30 or so submissions (of ~500) download before it errors:

2018-08-07 09:38:34,568 [Thread-13] ERROR o.o.briefcase.ui.export.ExportPanel - Error while exporting forms
org.opendatakit.briefcase.reused.BriefcaseException: org.xmlpull.v1.XmlPullParserException: unterminated entity ref (position:ENTITY_REF &?O????Sy@1:15 in java.io.InputStreamReader@71612a) 
	at org.opendatakit.briefcase.export.SubmissionParser.parse(SubmissionParser.java:203)
	at org.opendatakit.briefcase.export.SubmissionParser.decrypt(SubmissionParser.java:171)
	at org.opendatakit.briefcase.export.SubmissionParser.lambda$parseSubmission$5(SubmissionParser.java:142)
	at java.util.Optional.flatMap(Unknown Source)
	at org.opendatakit.briefcase.export.SubmissionParser.parseSubmission(SubmissionParser.java:122)
	at org.opendatakit.briefcase.export.ExportToCsv.lambda$export$3(ExportToCsv.java:109)
	at java.util.stream.ReferencePipeline$3$1.accept(Unknown Source)
	at java.util.ArrayList$ArrayListSpliterator.forEachRemaining(Unknown Source)
	at java.util.stream.AbstractPipeline.copyInto(Unknown Source)
	at java.util.stream.AbstractPipeline.wrapAndCopyInto(Unknown Source)
	at java.util.stream.ForEachOps$ForEachOp.evaluateSequential(Unknown Source)
	at java.util.stream.ForEachOps$ForEachOp$OfRef.evaluateSequential(Unknown Source)
	at java.util.stream.AbstractPipeline.evaluate(Unknown Source)
	at java.util.stream.ReferencePipeline.forEach(Unknown Source)
	at org.opendatakit.briefcase.export.ExportToCsv.export(ExportToCsv.java:112)
	at org.opendatakit.briefcase.ui.export.ExportPanel.lambda$export$10(ExportPanel.java:199)
	at java.util.stream.ForEachOps$ForEachOp$OfRef.accept(Unknown Source)
	at java.util.stream.ReferencePipeline$11$1.accept(Unknown Source)
	at java.util.ArrayList$ArrayListSpliterator.forEachRemaining(Unknown Source)
	at java.util.stream.AbstractPipeline.copyInto(Unknown Source)
	at java.util.stream.ForEachOps$ForEachTask.compute(Unknown Source)
	at java.util.concurrent.CountedCompleter.exec(Unknown Source)
	at java.util.concurrent.ForkJoinTask.doExec(Unknown Source)
	at java.util.concurrent.ForkJoinTask.doInvoke(Unknown Source)
	at java.util.concurrent.ForkJoinTask.invoke(Unknown Source)
	at java.util.stream.ForEachOps$ForEachOp.evaluateParallel(Unknown Source)
	at java.util.stream.ForEachOps$ForEachOp$OfRef.evaluateParallel(Unknown Source)
	at java.util.stream.AbstractPipeline.evaluate(Unknown Source)
	at java.util.stream.ReferencePipeline.forEach(Unknown Source)
	at org.opendatakit.briefcase.ui.export.ExportPanel.export(ExportPanel.java:188)
	at java.lang.Thread.run(Unknown Source)
Caused by: org.xmlpull.v1.XmlPullParserException: unterminated entity ref (position:ENTITY_REF &?O????Sy@1:15 in java.io.InputStreamReader@71612a) 
	at org.kxml2.io.KXmlParser.exception(Unknown Source)
	at org.kxml2.io.KXmlParser.error(Unknown Source)
	at org.kxml2.io.KXmlParser.pushEntity(Unknown Source)
	at org.kxml2.io.KXmlParser.nextImpl(Unknown Source)
	at org.kxml2.io.KXmlParser.nextToken(Unknown Source)
	at org.kxml2.kdom.Node.parse(Unknown Source)
	at org.kxml2.kdom.Document.parse(Unknown Source)
	at org.opendatakit.briefcase.export.SubmissionParser.parse(SubmissionParser.java:200)

Sorry for multiple messages - I've been able to re-create the error again on a similar form by using the 'forbidden' xml characters as entries in a survey (@, ? etc.) Is there any way to tell it to ignore these characters?

Hi, @Samuel_Watson!

Could you attach the blank form and the submission file that is producing the error so that I can add it to my tests?

The form data are encrypted so I can't retrieve the submission file - is there a way of doing it? Sorry if I'm missing something obvious.

Hi, @Samuel_Watson!

If you have pulled the form with Briefcase, you should have a directory named after the form in your storage directory. A zip of that directory would totally work.

If you don't want to zip everything (form and all submissions), you can take the xml file inside the form's directory and then you need to find the folder of the submission that is failing (identified with its UID) inside the instances folder.

Inside that second folder you should find a submission.xml and a submission.xml.enc files (the second is the encrypted one).

@ggalmazor, sorry again for the multiple messages! I've also attached here the submission.xml of the first instance where it fails in the proper data, although I can't send the actual submission unfortunately, so I don't know if that'll help. I'll keep trying to create a reproducible example of the exact error
submission.xml (1.3 KB)

EDIT - I deleted the post above as it was misleading: an identical form to the one that's failing is working fine with all my test submissions so far. Of the 500 or so submissions on the Aggregate server, it seems about half cause the error in the OP, and I have not idea what is different about them. All submissions have osm file attachments (usually more than one) and a jpg. I'm not sure how to debug this as all previous testing didn't show this error and now we have time-sensitive data stuck on the server - I can send anything but the data. Is there a way to access and decrypt the data without Briefcase?

@Samuel_Watson Can you also attach the XLSForm/XForm that you used? And also try much older version (e.g., v1.4.9) and see if that works.

Hi @yanokwa, the old version worked! Phew! Thank you. This is the xform used Nigeria__Ground_Truthing_Questionnaire.xml (152.6 KB)

Glad it worked, so now we know it's a regression, @ggalmazor!

@Samuel_Watson, can you try one more thing? I think this is a UTF-8 encoding issue and we have a tweaked version of Briefcase at briefcase-utf8.jar. Can you try that and see if that also works?

@yanokwa it threw an error at 2% but kept going to 25% then stopped exporting (the behaviour in the v1.11 released version was that it would just error at 5% and stop). However, the log doesn't seem to be catching anything from the version in your last reply so I can't report back what error in particular.

1 Like

Hi, @Samuel_Watson! Thanks for all your feedback. It helps a lot.

Can you try with v1.11.3, please? https://github.com/opendatakit/briefcase/releases/tag/v1.11.3

Hi @ggalmazor same issue as before unfortunately:

2018-08-09 12:28:36,854 [Thread-8] ERROR o.o.briefcase.ui.export.ExportPanel - Error while exporting forms
org.opendatakit.briefcase.reused.BriefcaseException: org.xmlpull.v1.XmlPullParserException: unterminated entity ref (position:ENTITY_REF &?O????Sy@1:15 in java.io.InputStreamReader@19e254d) 
	at org.opendatakit.briefcase.export.SubmissionParser.parse(SubmissionParser.java:203)
	at org.opendatakit.briefcase.export.SubmissionParser.decrypt(SubmissionParser.java:171)
	at org.opendatakit.briefcase.export.SubmissionParser.lambda$parseSubmission$5(SubmissionParser.java:142)
	at java.util.Optional.flatMap(Unknown Source)
	at org.opendatakit.briefcase.export.SubmissionParser.parseSubmission(SubmissionParser.java:122)
	at org.opendatakit.briefcase.export.ExportToCsv.lambda$export$3(ExportToCsv.java:109)
	at java.util.stream.ReferencePipeline$3$1.accept(Unknown Source)
	at java.util.ArrayList$ArrayListSpliterator.forEachRemaining(Unknown Source)
	at java.util.stream.AbstractPipeline.copyInto(Unknown Source)
	at java.util.stream.AbstractPipeline.wrapAndCopyInto(Unknown Source)
	at java.util.stream.ForEachOps$ForEachOp.evaluateSequential(Unknown Source)
	at java.util.stream.ForEachOps$ForEachOp$OfRef.evaluateSequential(Unknown Source)
	at java.util.stream.AbstractPipeline.evaluate(Unknown Source)
	at java.util.stream.ReferencePipeline.forEach(Unknown Source)
	at org.opendatakit.briefcase.export.ExportToCsv.export(ExportToCsv.java:112)
	at org.opendatakit.briefcase.ui.export.ExportPanel.lambda$export$10(ExportPanel.java:199)
	at java.util.stream.ForEachOps$ForEachOp$OfRef.accept(Unknown Source)
	at java.util.stream.ReferencePipeline$11$1.accept(Unknown Source)
	at java.util.ArrayList$ArrayListSpliterator.forEachRemaining(Unknown Source)
	at java.util.stream.AbstractPipeline.copyInto(Unknown Source)
	at java.util.stream.ForEachOps$ForEachTask.compute(Unknown Source)
	at java.util.concurrent.CountedCompleter.exec(Unknown Source)
	at java.util.concurrent.ForkJoinTask.doExec(Unknown Source)
	at java.util.concurrent.ForkJoinTask.doInvoke(Unknown Source)
	at java.util.concurrent.ForkJoinTask.invoke(Unknown Source)
	at java.util.stream.ForEachOps$ForEachOp.evaluateParallel(Unknown Source)
	at java.util.stream.ForEachOps$ForEachOp$OfRef.evaluateParallel(Unknown Source)
	at java.util.stream.AbstractPipeline.evaluate(Unknown Source)
	at java.util.stream.ReferencePipeline.forEach(Unknown Source)
	at org.opendatakit.briefcase.ui.export.ExportPanel.export(ExportPanel.java:188)
	at java.lang.Thread.run(Unknown Source)
Caused by: org.xmlpull.v1.XmlPullParserException: unterminated entity ref (position:ENTITY_REF &?O????Sy@1:15 in java.io.InputStreamReader@19e254d) 
	at org.kxml2.io.KXmlParser.exception(Unknown Source)
	at org.kxml2.io.KXmlParser.error(Unknown Source)
	at org.kxml2.io.KXmlParser.pushEntity(Unknown Source)
	at org.kxml2.io.KXmlParser.nextImpl(Unknown Source)
	at org.kxml2.io.KXmlParser.nextToken(Unknown Source)
	at org.kxml2.kdom.Node.parse(Unknown Source)
	at org.kxml2.kdom.Document.parse(Unknown Source)
	at org.opendatakit.briefcase.export.SubmissionParser.parse(SubmissionParser.java:200)

Hi, @Samuel_Watson!

I've tried your form and the one @knicholas is having the same issue with. I've changed the form to use my public encryption key and I've created some submissions to test it, but I haven't been able to reproduce the problem yet.

Samuel, you mentioned that you were able to reproduce the error by introducing special characters like @, and ? in the answers. Does that still hold true? @knicholas, does that relate to your situation?

Also, @Samuel_Watson, you mentioned you were able to decrypt one of the submissions with Briefcase v1.4.9. Could you send me the decrypted csv from a submission that would fail with the current v1.11.3? That could give me hints about the kind of content that makes Briefcase fail.

To use the submissions you have sent me, I would have to use your privatekey, which I would understand if you don't want to share with me.

Unfortunately I may have just confused things by talking about special characters. They form I sent also works fine on all my tests - it's only appeared once we have a large number of submissions. Indeed the same error occurred with a different form we had 1100 submissions for. However I can't replicate the error on a duplicate form and a small number of submissions. Looking at the data I can't see at all what is different about the ones where it just stops.

OK, thanks for the feedback, @Samuel_Watson!

I wanted to follow up on this, @Samuel_Watson. We'd like to get a reproducible case so we can fix this bug!

It sounds like you have a form and a large number of submissions that fails with v1.11.3. And that same form and submissions work with v1.4.9. Is that correct?

Would you be able to share your Briefcase Storage folder and your private key with @ggalmazor? Please don't upload them here, but instead, send him an email directly at ggalmazor@gmail.com and he can help coordinate a secure way to transfer those files.