Problems with exporting to .csv via CLI

Hi ODK team,

I'm really enjoying using ODK -- super useful set of tools. Thank you. I am fairly new to the world of ODK (and programming in R and doing anything via CLI) but I am learning a lot.

Hopefully this explains my problem, but if something isn't clear I can try to give you more information.

1. What is the problem? Be very detailed.

I am trying to export forms using CLI (actually in R) and I keep getting an error. FWIW, I don't actually need the csv, but I think getting the csv first is probably the easiest way to create a data frame in R from my data?

Firstly I pull the forms using;

system("java -jar odkBriefcase_latest.jar -plla -U {URL here} -u {XXXX} -p {XXX} -sd THIS -id test_drink")

This runs for a while and - admittedly - ends in an error BUT the storage directory is created abd as far as I can work out all the submissions are actually pulled and are now on my computer in .xml . So I don't THINK this error matters?

The error is;
Error pulling a form: java.nio.file.NoSuchFileException: THIS/ODK Briefcase Storage/THIS/ODK Briefcase Storage/forms/test_drink/metadata.json (see the logs for more info)

Then I try to export the submissions using the following;

system("java -jar odkBriefcase_latest.jar -e -id test_drink -sd THIS -ed THAT -ef test_drink.csv")

But this comes straight back with an error( and the THAT export directory isn't created);

Unexpected error in Briefcase. Please review briefcase.log for more information. For help, post to https://forum.getodk.org/c/support

When I look at the logs the output is very long but I think the salient bit where it goes wrong is this bit;

2019-10-16 20:36:34,706 [main] ERROR org.opendatakit.briefcase.Launcher - Error
java.io.UncheckedIOException: java.nio.file.NoSuchFileException: THIS/ODK Briefcase Storage/THIS/ODK Briefcase Storage/forms/test_drink/metadata.json

It looks to me (although I don't really know) that it's looking for a storage directory called ~THIS/ODK Briefcase Storage/THIS/ODK Briefcase Storage ..... and presumably the storage directory is just ~THIS/ODK Briefcase Storage (without the second bit).

2. What app or server are you using and on what device and operating system? Include version numbers.

ODK aggregate server
Briefcase 1.17.0
Java Runtime Environment 1.8.221.11
Mac OS Mojave 10.14.6
RStudio 1.2.1335

3. What you have you tried to fix the problem?

I don't really have any ideas! I've updated to the latest versions of everything.

I've read through the documentation at https://docs.opendatakit.org/briefcase-using/

I think it's a problem with the export rather than the pull -- although given that the pull also returns an error (although seems to also work fine) I'm not sure.

4. What steps can we take to reproduce the problem?

Whilst this is all test data, I don't think I can give the server username and password because there's some other people's stuff on there (I'm using a server we use at LSHTM for training).

5. Anything else we should know or have? If you have a test form or screenshots or logs, attach below.

Screenshots are attached
46|689x465

1 Like

O, the other thing I tried to fix the problem was to create a folder called THAT (I wasn't sure if the CLI command would create the export directory, or if it expected the export directory it to exist already).
But it didn't make any difference - the error was the same.

1 Like

I may need to do something similar in a project I'm working on, but I'm not quite at that point yet. I've bookmarked another post I saw here on the forum, maybe it could point you in the right direction (I haven't tried it yet). It's a post from user @chrissyhroberts
Run an external script via ODK briefcase. (check the last post, with a link to his code on github)

Hi @rachaelb

Try making a clean folder, then use the GUI to create your ODK Briefcase Storage folder.
If all is good, then you should be able to see a metadata.json file in each form's folder in the storage directory (see image)

This file tells Briefcase where it left off with exports and pulls, which is how the 'start-from-last' and 'smart-append' features work. I think that the error you are seeing is happening because your version of aggregate. I don't think that the CLI is able to make this file if missing at present.

After you've used the GUI the first time, then using the CLI should become possible.

This issue has already been reported here https://github.com/opendatakit/briefcase/issues/824 and hopefully will be fixed soon.

If issue persists, maybe go back to briefcase v.1.15 or v.1.16 as I think both of these preceded the change that's introduced this issue.

Hi!

I've just found that this issue is related to using relative paths in the -sd arg. Could you try to use absolute paths? and check if you're still getting errors?

I'm working on a fix to be able to use relatve paths.

Would you try this jar to see if the error persist, @rachaelb?

Thanks for the replies everyone. Really appreciate it. Internet is not happy at all presently at my house (there is a rumour that there may or may not be a problem with the cable coming into the whole country...). I'll try these suggestions / the new jar file tomorrow in work and let you know how I get on.

1 Like

Thank you everyone. @ggalmazor that new jar worked perfectly, even with relative file paths. Thank you!

3 Likes

Thanks for reporting back, @rachaelb. We will be releasing a fixed version of Briefcase soon.

1 Like

Hi,

I seem to have the same problem. I am using Briefcase 1.17.2. See attached error. I have been using version 1.17 but just tried out the latest version based on the feedback above.Error.txt (14.7 KB)

Paul