Using cURL to download CSV file?

Support
1

Hi All!

I am using a Google ODK Aggregate Server (any Google account can see
it): http://moreeyestest.appspot.com/

I'd like to script the downloading of the CSV file with the results.
I've found a bit of documentation on how to do it, but something just
isn't working. I was hoping someone could offer some advice.

I've used these sites:

And come up with the following procedure (running in bash on Linux):

  1. AUTH=$(curl https://www.google.com/accounts/ClientLogin -d
    Email=email -d Passwd=passwd -d accountType=GOOGLE -d source=Google-
    cURL-Example -d service=apps | cut -d= -f2) which returns the
    variable $AUTH that looks like "DQAAAKQAAADccA6mq..."
  2. curl http://moreeyestest.appspot.com/csv?odkId=more_eyes_test --
    header "Authorization: GoogleLogin $AUTH"

The first step appears to be working, as I get some form of
authentication string back. When the second command is run passing in
that authentication string, I always get back the same HTML formatted
error telling me that a Login is required:

Open Data Kit Aggregate

Open Data Kit Aggregate



Login Required

Please log in

I thought it might be the fact that I wasn't using the right service
name, so I used the second link I mentioned above and tried each one
with no success. I've also tried the string "ah" as mentioned in the
returned HTML error posted above.

I'm completely stumped. Does anyone have any suggestions on something
else to try? Thanks in advance for any help.

-Bill

2

Bill,

Could you add a -v for verbose to your two curl commands and paste the result into pastie or dpaste for us to see?

Thanks,
—Clint

··· On Dec 10, 2010, at 9:09 AM, Bill Wacek wrote:

Hi All!

I am using a Google ODK Aggregate Server (any Google account can see
it): http://moreeyestest.appspot.com/

I'd like to script the downloading of the CSV file with the results.
I've found a bit of documentation on how to do it, but something just
isn't working. I was hoping someone could offer some advice.

I've used these sites:
http://code.google.com/apis/gdata/articles/using_cURL.html
http://code.google.com/apis/gdata/faq.html#Authentication

And come up with the following procedure (running in bash on Linux):

  1. AUTH=$(curl https://www.google.com/accounts/ClientLogin -d
    Email=email -d Passwd=passwd -d accountType=GOOGLE -d source=Google-
    cURL-Example -d service=apps | cut -d= -f2) which returns the
    variable $AUTH that looks like "DQAAAKQAAADccA6mq..."
  2. curl http://moreeyestest.appspot.com/csv?odkId=more_eyes_test --
    header "Authorization: GoogleLogin $AUTH"

The first step appears to be working, as I get some form of
authentication string back. When the second command is run passing in
that authentication string, I always get back the same HTML formatted
error telling me that a Login is required:

Open Data Kit Aggregate

Open Data Kit Aggregate



Login Required

Please log in

I thought it might be the fact that I wasn't using the right service
name, so I used the second link I mentioned above and tried each one
with no success. I've also tried the string "ah" as mentioned in the
returned HTML error posted above.

I'm completely stumped. Does anyone have any suggestions on something
else to try? Thanks in advance for any help.

-Bill

--
Post: opendatakit@googlegroups.com
Unsubscribe: opendatakit+unsubscribe@googlegroups.com
Options: http://groups.google.com/group/opendatakit?hl=en

3

Sure. I had never seen pastie before, very useful site!

http://pastie.org/1365986

I trimmed the security keys in the latter command result. I was
surprised to see the three keys in that result. The first one I
understand is me sending the key, but I don't understand why two more
are being sent. Those two extra keys are identical up to around 60
characters or so, then they split off.

-Bill

··· On Dec 10, 12:21 pm, Clint Tseng wrote: > Bill, > > Could you add a -v for verbose to your two curl commands and paste the result into pastie or dpaste for us to see? > > Thanks, > —Clint > > On Dec 10, 2010, at 9:09 AM, Bill Wacek wrote: > > > > > > > > > Hi All! > > > I am using a Google ODK Aggregate Server (any Google account can see > > it): http://moreeyestest.appspot.com/ > > > I'd like to script the downloading of the CSV file with the results. > > I've found a bit of documentation on how to do it, but something just > > isn't working. I was hoping someone could offer some advice. > > > I've used these sites: > >http://code.google.com/apis/gdata/articles/using_cURL.html > >http://code.google.com/apis/gdata/faq.html#Authentication > > > And come up with the following procedure (running in bash on Linux): > > > 1. AUTH=$(curlhttps://www.google.com/accounts/ClientLogin-d > > Email=email -d Passwd=passwd -d accountType=GOOGLE -d source=Google- > > cURL-Example -d service=apps | cut -d\= -f2) which returns the > > variable $AUTH that looks like "DQAAAKQAAADccA6mq..." > > 2. curlhttp://moreeyestest.appspot.com/csv?odkId=more_eyes_test-- > > header "Authorization: GoogleLogin $AUTH" > > > The first step appears to be working, as I get some form of > > authentication string back. When the second command is run passing in > > that authentication string, I always get back the same HTML formatted > > error telling me that a Login is required: > > > Open Data Kit Aggregate > head>

Open Data Kit Aggregate

> FONT>

Login Required

Please log > > in

> > > I thought it might be the fact that I wasn't using the right service > > name, so I used the second link I mentioned above and tried each one > > with no success. I've also tried the string "ah" as mentioned in the > > returned HTML error posted above. > > > I'm completely stumped. Does anyone have any suggestions on something > > else to try? Thanks in advance for any help. > > > -Bill > > > -- > > Post: opendatakit@googlegroups.com > > Unsubscribe: opendatakit+unsubscribe@googlegroups.com > > Options:http://groups.google.com/group/opendatakit?hl=en
4

Try adding

| grep "^Auth"

in the pipechain before your cut command?

—Clint

··· On Dec 10, 2010, at 10:45 AM, Bill Wacek wrote:

Sure. I had never seen pastie before, very useful site!

http://pastie.org/1365986

I trimmed the security keys in the latter command result. I was
surprised to see the three keys in that result. The first one I
understand is me sending the key, but I don't understand why two more
are being sent. Those two extra keys are identical up to around 60
characters or so, then they split off.

-Bill

On Dec 10, 12:21 pm, Clint Tseng c...@cs.washington.edu wrote:

Bill,

Could you add a -v for verbose to your two curl commands and paste the result into pastie or dpaste for us to see?

Thanks,
—Clint

On Dec 10, 2010, at 9:09 AM, Bill Wacek wrote:

Hi All!

I am using a Google ODK Aggregate Server (any Google account can see
it): http://moreeyestest.appspot.com/

I'd like to script the downloading of the CSV file with the results.
I've found a bit of documentation on how to do it, but something just
isn't working. I was hoping someone could offer some advice.

I've used these sites:
http://code.google.com/apis/gdata/articles/using_cURL.html
http://code.google.com/apis/gdata/faq.html#Authentication

And come up with the following procedure (running in bash on Linux):

  1. AUTH=$(curlhttps://www.google.com/accounts/ClientLogin-d
    Email=email -d Passwd=passwd -d accountType=GOOGLE -d source=Google-
    cURL-Example -d service=apps | cut -d= -f2) which returns the
    variable $AUTH that looks like "DQAAAKQAAADccA6mq..."
  2. curlhttp://moreeyestest.appspot.com/csv?odkId=more_eyes_test--
    header "Authorization: GoogleLogin $AUTH"

The first step appears to be working, as I get some form of
authentication string back. When the second command is run passing in
that authentication string, I always get back the same HTML formatted
error telling me that a Login is required:

Open Data Kit Aggregate

Open Data Kit Aggregate



Login Required

Please log in

I thought it might be the fact that I wasn't using the right service
name, so I used the second link I mentioned above and tried each one
with no success. I've also tried the string "ah" as mentioned in the
returned HTML error posted above.

I'm completely stumped. Does anyone have any suggestions on something
else to try? Thanks in advance for any help.

-Bill

--
Post: opendatakit@googlegroups.com
Unsubscribe: opendatakit+unsubscribe@googlegroups.com
Options:http://groups.google.com/group/opendatakit?hl=en

--
Post: opendatakit@googlegroups.com
Unsubscribe: opendatakit+unsubscribe@googlegroups.com
Options: http://groups.google.com/group/opendatakit?hl=en

5

Well that certainly points out one error I was making. I forgot there
were three keys being sent back, separated by spaces. Doh!

However, it's still not letting me in:

http://pastie.org/1366206

I got another idea from one if the sites I saw. I left in the "Auth="
part of the key instead of stripping it out. Some other examples had
that, but it didn't work for me either. That attempt and result is
shown at the bottom of that pastie page.

-Bill

··· On Dec 10, 2:06 pm, Clint Tseng wrote: > Try adding > > | grep "^Auth" > > in the pipechain before your cut command? > > —Clint > > On Dec 10, 2010, at 10:45 AM, Bill Wacek wrote: > > > > > > > > > Sure. I had never seen pastie before, very useful site! > > >http://pastie.org/1365986 > > > I trimmed the security keys in the latter command result. I was > > surprised to see the three keys in that result. The first one I > > understand is me sending the key, but I don't understand why two more > > are being sent. Those two extra keys are identical up to around 60 > > characters or so, then they split off. > > > -Bill > > > On Dec 10, 12:21 pm, Clint Tseng wrote: > >> Bill, > > >> Could you add a -v for verbose to your two curl commands and paste the result into pastie or dpaste for us to see? > > >> Thanks, > >> —Clint > > >> On Dec 10, 2010, at 9:09 AM, Bill Wacek wrote: > > >>> Hi All! > > >>> I am using a Google ODK Aggregate Server (any Google account can see > >>> it): http://moreeyestest.appspot.com/ > > >>> I'd like to script the downloading of the CSV file with the results. > >>> I've found a bit of documentation on how to do it, but something just > >>> isn't working. I was hoping someone could offer some advice. > > >>> I've used these sites: > >>>http://code.google.com/apis/gdata/articles/using_cURL.html > >>>http://code.google.com/apis/gdata/faq.html#Authentication > > >>> And come up with the following procedure (running in bash on Linux): > > >>> 1. AUTH=$(curlhttps://www.google.com/accounts/ClientLogin-d > >>> Email=email -d Passwd=passwd -d accountType=GOOGLE -d source=Google- > >>> cURL-Example -d service=apps | cut -d\= -f2) which returns the > >>> variable $AUTH that looks like "DQAAAKQAAADccA6mq..." > >>> 2. curlhttp://moreeyestest.appspot.com/csv?odkId=more_eyes_test-- > >>> header "Authorization: GoogleLogin $AUTH" > > >>> The first step appears to be working, as I get some form of > >>> authentication string back. When the second command is run passing in > >>> that authentication string, I always get back the same HTML formatted > >>> error telling me that a Login is required: > > >>> Open Data Kit Aggregate >>> head>

Open Data Kit Aggregate

>>> FONT>

Login Required

Please log > >>> in

> > >>> I thought it might be the fact that I wasn't using the right service > >>> name, so I used the second link I mentioned above and tried each one > >>> with no success. I've also tried the string "ah" as mentioned in the > >>> returned HTML error posted above. > > >>> I'm completely stumped. Does anyone have any suggestions on something > >>> else to try? Thanks in advance for any help. > > >>> -Bill > > >>> -- > >>> Post: opendatakit@googlegroups.com > >>> Unsubscribe: opendatakit+unsubscribe@googlegroups.com > >>> Options:http://groups.google.com/group/opendatakit?hl=en > > > -- > > Post: opendatakit@googlegroups.com > > Unsubscribe: opendatakit+unsubscribe@googlegroups.com > > Options:http://groups.google.com/group/opendatakit?hl=en
6

The Briefcase applet, available after Aggregate 0.9.3, will download your
data into a CSV file. You should look at that before scripting new cUrl
code.

If you do still need Curl, you should look at the /csvFragment servlet page
(it displays help information). That is what the Briefcase code uses and it
provides xml-wrapped fragments of csv files, supports passing a cursor back
and forth for follow-on requests, etc.

This format will change somewhat for Aggregate 1.x (still being coded).

Mitch

··· On Fri, Dec 10, 2010 at 11:47 AM, Bill Wacek wrote:

Well that certainly points out one error I was making. I forgot there
were three keys being sent back, separated by spaces. Doh!

However, it's still not letting me in:

http://pastie.org/1366206

I got another idea from one if the sites I saw. I left in the "Auth="
part of the key instead of stripping it out. Some other examples had
that, but it didn't work for me either. That attempt and result is
shown at the bottom of that pastie page.

-Bill

On Dec 10, 2:06 pm, Clint Tseng c...@cs.washington.edu wrote:

Try adding

| grep "^Auth"

in the pipechain before your cut command?

—Clint

On Dec 10, 2010, at 10:45 AM, Bill Wacek wrote:

Sure. I had never seen pastie before, very useful site!

http://pastie.org/1365986

I trimmed the security keys in the latter command result. I was
surprised to see the three keys in that result. The first one I
understand is me sending the key, but I don't understand why two more
are being sent. Those two extra keys are identical up to around 60
characters or so, then they split off.

-Bill

On Dec 10, 12:21 pm, Clint Tseng c...@cs.washington.edu wrote:

Bill,

Could you add a -v for verbose to your two curl commands and paste the
result into pastie or dpaste for us to see?

Thanks,
—Clint

On Dec 10, 2010, at 9:09 AM, Bill Wacek wrote:

Hi All!

I am using a Google ODK Aggregate Server (any Google account can see
it): http://moreeyestest.appspot.com/

I'd like to script the downloading of the CSV file with the results.
I've found a bit of documentation on how to do it, but something just
isn't working. I was hoping someone could offer some advice.

I've used these sites:
http://code.google.com/apis/gdata/articles/using_cURL.html
http://code.google.com/apis/gdata/faq.html#Authentication

And come up with the following procedure (running in bash on Linux):

  1. AUTH=$(curlhttps://www.google.com/accounts/ClientLogin-d
    Email=email -d Passwd=passwd -d accountType=GOOGLE -d source=Google-
    cURL-Example -d service=apps | cut -d= -f2) which returns the
    variable $AUTH that looks like "DQAAAKQAAADccA6mq..."
  2. curlhttp://moreeyestest.appspot.com/csv?odkId=more_eyes_test--
    header "Authorization: GoogleLogin $AUTH"

The first step appears to be working, as I get some form of
authentication string back. When the second command is run passing
in
that authentication string, I always get back the same HTML formatted
error telling me that a Login is required:

Open Data Kit Aggregate

Open Data Kit Aggregate



Login Required

Please www.google.com/accounts/ServiceLogin?service=ah&continue=http://moree.....
.">log

in

I thought it might be the fact that I wasn't using the right service
name, so I used the second link I mentioned above and tried each one
with no success. I've also tried the string "ah" as mentioned in the
returned HTML error posted above.

I'm completely stumped. Does anyone have any suggestions on
something
else to try? Thanks in advance for any help.

-Bill

--
Post: opendatakit@googlegroups.com
Unsubscribe: opendatakit+unsubscribe@googlegroups.comopendatakit%2Bunsubscribe@googlegroups.com
Options:http://groups.google.com/group/opendatakit?hl=en

--
Post: opendatakit@googlegroups.com
Unsubscribe: opendatakit+unsubscribe@googlegroups.comopendatakit%2Bunsubscribe@googlegroups.com
Options:http://groups.google.com/group/opendatakit?hl=en

--
Post: opendatakit@googlegroups.com
Unsubscribe: opendatakit+unsubscribe@googlegroups.comopendatakit%2Bunsubscribe@googlegroups.com
Options: http://groups.google.com/group/opendatakit?hl=en

--
Mitch Sundt
Software Engineer
http://www.OpenDataKit.org
University of Washington
mitchellsundt@gmail.com