Problem with ODK aggregate and Google Fusion Tables or Spreadsheets

David · March 31, 2011, 2:00pm

Dear all,

Any help that can be offered would be much appreciated. When I try
to set up an ODK Aggregate (v.0.95) connection to Google Fusion
tables, the error below is returned. I've tried shortening field
names, changing survey IDs, and nothing seems to help. ODK can set
up a connection to Google Spreadsheets, but no data are transmitted
other than submission times.

What is the problem? The XML is attached for reference.

Thanks in advance.

Best,
David

adoption-survey.xml (102 KB)

Mitch_Sundt · March 31, 2011, 6:00pm

Hi David,

We are passing the sql=CREATE... parameter as an argument on the POST
request's URL, instead of in the body of the request, and that is making the
URL string too long. It looks like FusionTables supports passing the
parameter in the body of the POST request, which should eliminate this
limitation.

I'll see what we can do to change this.

Mitch

···

On Thu, Mar 31, 2011 at 7:00 AM, David A. Raitzer wrote:

Dear all,

Any help that can be offered would be much appreciated. When I try to set up an ODK Aggregate (v.0.95) connection to Google Fusion tables, the error below is returned. I've tried shortening field names, changing survey IDs, and nothing seems to help. ODK can set up a connection to Google Spreadsheets, but no data are transmitted other than submission times.

What is the problem? The XML is attached for reference.

Thanks in advance.

Best,
David

Uncaught exception from servlet
java.net.MalformedURLException: Invalid URL: http://tables.googlelabs.com/api/query?sql=CREATE+TABLE+'AdoptionSurvey'+('RBPsourcetechnologyintroduction'%3Astring%2C'NCarea'%3Anumber%2C'surveyordetailscollectortype'%3Astring%2C'IIWMirrigationfrequency'%3Astring%2C'locationfieldslocationtype'%3Astring%2C'DSRsourcetechnologyintroduction'%3Astring%2C'IIWMguide'%3Astring%2C'village'%3Astring%2C'IWMstaleseedbed'%3Astring%2C'ICarea'%3Anumber%2C'locationAltitude'%3Alocation%2C'SPinitialyear'%3Astring%2C'ICinitialyear'%3Astring%2C'RBPwidth'%3Astring%2C'RTtype'%3Astring%2C'RBPresidueretentionpreviouscrop'%3Astring%2C'CSDintroducedcrop'%3Astring%2C'DSRirrigation'%3Astring%2C'preDSRpractice'%3Astring%2C'LLLarea'%3Anumber%2C'manuallatitude'%3Anumber%2C'RCsourcetechnologyintroduction'%3Astring%2C'INMliming'%3Astring%2C'IIWMcrop'%3Astring%2C'RTmaincroppingsystem'%3Astring%2C'RCmaincrops'%3Astring%2C'ICmaincroppingsystem'%3Astring%2C'INMothernutrients'%3Astring%2C'UPTRmaincroppingsystem'%3Astring%2C'comment'%3Astring%2C'IIWMinitialyear'%3Astring%2C'farmarea'%3Anumber%2C'IIWMsourcetechnologyintroduction'%3Astring%2C'LLLmaincroppingsystem'%3Astring%2C'ZTmaincroppingsystem'%3Astring%2C'IWMUPTR'%3Astring%2C'IPHSinitialyear'%3Astring%2C'RCmaincroppingsystem'%3Astring%2C'LLLZT'%3Astring%2C'ZTinitialyear'%3Astring%2C'IWMZT'%3Astring%2C'CSDreplacedcrop'%3Astring%2C'RBPtype'%3Astring%2C'DSRirrigationscheduling'%3Astring%2C'locationAccuracy'%3Alocation%2C'adoptedtechnologies'%3Astring%2C'IPHSstoragetype'%3Astring%2C'surveyordetailssurveymethod'%3Astring%2C'IWMherbicides'%3Astring%2C'INMarea'%3Anumber%2C'RBParea'%3Anumber%2C'district'%3Astring%2C'IWMotherweedmanagement'%3Astring%2C'IPHSstoragecapacity'%3Anumber%2C'IWMbrownmanuring'%3Astring%2C'SubmissionDate'%3Adatetime%2C'LLLRT'%3Astring%2C'CFpreviouscropfallowperiod'%3Astring%2C'DSRarea'%3Anumber%2C'IWMRT'%3Astring%2C'INMguide'%3Astring%2C'farmerdetailsnamefarmer'%3Astring%2C'INMnutrients'%3Astring%2C'IWMsourcetechnologyintroduction'%3Astring%2C'LLLsourcetechnologyintroduction'%3Astring%2C'RTsourcetechnologyintroduction'%3Astring%2C'telephone'%3Anumber%2C'NCsourcetechnologyintroduction'%3Astring%2C'SPcrop'%3Astring%2C'location'%3Alocation%2C'percentagefarmarearented'%3Astring%2C'ZTarea'%3Anumber%2C'INMNPKincrease'%3Astring%2C'RBPmaincroppingsystem'%3Astring%2C'CFseasonnewcrop'%3Astring%2C'IIWMchangedirrigationtiming'%3Astring%2C'IWMherbicideapplicationfrequency'%3Astring%2C'CFinitialyear'%3Astring%2C'DSRmethod'%3Astring%2C'ICintercrop'%3Astring%2C'ICmaincrops'%3Astring%2C'CSDmaincroppingsystem'%3Astring%2C'IWMarea'%3Anumber%2C'UPTRarea'%3Anumber%2C'IWMDSR'%3Astring%2C'RTarea'%3Anumber%2C'NColdvariety'%3Astring%2C'RCarea'%3Anumber%2C'IPHSsourcetechnologyintroduction'%3Astring%2C'NCinitialyear'%3Astring%2C'DSRinitialyear'%3Astring%2C'INMsplitnitrogenapplication'%3Astring%2C'UPTRRT'%3Astring%2C'NCDSR'%3Astring%2C'CSDarea'%3Anumber%2C'RBPcrop'%3Astring%2C'RCinitialyear'%3Astring%2C'UPTRZT'%3Astring%2C'ZTcrop'%3Astring%2C'IWMinitialyear'%3Astring%2C'IIWMDSR'%3Astring%2C'ICsourcetechnologyintroduction'%3Astring%2C'SPseedtreatment'%3Astring%2C'preUPTRpractice'%3Astring%2C'ZTresidueretentionpreviouscrop'%3Astring%2C'INMbandedfertilizerapplication'%3Astring%2C'IIWMarea'%3Anumber%2C'RTcrop'%3Astring%2C'LLLlastyear'%3Astring%2C'NCZT'%3Astring%2C'CFsourcetechnologyintroduction'%3Astring%2C'ZTsourcetechnologyintroduction'%3Astring%2C'IWMRBP'%3Astring%2C'INMNPKdecrease'%3Astring%2C'RTinitialyear'%3Astring%2C'CSDinitialyear'%3Astring%2C'farmerdetailsfarmergender'%3Astring%2C'RTpriortillagepasses'%3Anumber%2C'NCnewvariety'%3Astring%2C'DSRmaincroppingsystem'%3Astring%2C'UPTRLLL'%3Astring%2C'NCcrop'%3Astring%2C'INMureagranulesplacement'%3Astring%2C'ZTpriortillagepasses'%3Anumber%2C'DSRZT'%3Astring%2C'DSRLLL'%3Astring%2C'RBPinitialyear'%3Astring%2C'manuallongitude'%3Anumber%2C'RTnumbertillages'%3Anumber%2C'NCRT'%3Astring%2C'UPTRtransplantingtype'%3Astring%2C'UPTRsourcetechnologyintroduction'%3Astring%2C'UPTRinitialyear'%3Astring%2C'IIWMUPTR'%3Astring%2C'RTresidueretentionpreviouscrop'%3Astring%2C'DSRRT'%3Astring%2C'INMinitialyear'%3Astring%2C'CSDsourcetechnologyintroduction'%3Astring%2C'INMsourcetechnologyintroduction'%3Astring%2C'IWMcrop'%3Astring%2C'INMcrop'%3Astring)

at com.google.appengine.api.urlfetch.URLFetchServiceImpl.convertApplicationException(URLFetchServiceImpl.java:106)
at com.google.appengine.api.urlfetch.URLFetchServiceImpl.fetch(URLFetchServiceImpl.java:41)
at com.google.apphosting.utils.security.urlfetch.URLFetchServiceStreamHandler$Connection.fetchResponse(URLFetchServiceStreamHandler.java:418)

at com.google.apphosting.utils.security.urlfetch.URLFetchServiceStreamHandler$Connection.getInputStream(URLFetchServiceStreamHandler.java:297)
at org.odk.aggregate.form.remoteserver.FusionTableOAuth.executeInsert(FusionTableOAuth.java:149)

at org.odk.aggregate.servlet.FusionTableServlet.doGet(FusionTableServlet.java:140)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:617)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:717)

at org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:511)
at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1166)
at com.google.apphosting.utils.servlet.ParseBlobUploadFilter.doFilter(ParseBlobUploadFilter.java:97)

at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1157)
at com.google.apphosting.runtime.jetty.SaveSessionFilter.doFilter(SaveSessionFilter.java:35)
at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1157)

at com.google.apphosting.utils.servlet.TransactionCleanupFilter.doFilter(TransactionCleanupFilter.java:43)
at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1157)
at org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:388)

at org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216)
at org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:182)
at org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:765)

at org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:418)
at com.google.apphosting.runtime.jetty.AppVersionHandlerMap.handle(AppVersionHandlerMap.java:238)
at org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152)

at org.mortbay.jetty.Server.handle(Server.java:326)
at org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:542)
at org.mortbay.jetty.HttpConnection$RequestHandler.headerComplete(HttpConnection.java:923)

at com.google.apphosting.runtime.jetty.RpcRequestParser.parseAvailable(RpcRequestParser.java:76)
at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:404)
at com.google.apphosting.runtime.jetty.JettyServletEngineAdapter.serviceRequest(JettyServletEngineAdapter.java:135)

at com.google.apphosting.runtime.JavaRuntime.handleRequest(JavaRuntime.java:261)
at com.google.apphosting.base.RuntimePb$EvaluationRuntime$2.handleRequest(RuntimePb.java:9285)
at com.google.net.rpc.impl.RpcUtil.runRpcInApplication(RpcUtil.java:437)

at com.google.net.rpc.impl.Server$RpcTask.runInContext(Server.java:573)
at com.google.tracing.TraceContext$TraceContextRunnable$1.run(TraceContext.java:448)
at com.google.tracing.TraceContext.runInContext(TraceContext.java:688)

at com.google.tracing.TraceContext$AbstractTraceContextCallback.runInInheritedContextNoUnref(TraceContext.java:326)
at com.google.tracing.TraceContext$AbstractTraceContextCallback.runInInheritedContext(TraceContext.java:318)

Unexpected exception from servlet: java.net.MalformedURLException: Invalid URL: http://tables.googlelabs.com/api/query?sql=CREATE+TABLE+'AdoptionSurvey'+('RBPsourcetechnologyintroduction'%3Astring%2C'NCarea'%3Anumber%2C'surveyordetailscollectortype'%3Astring%2C'IIWMirrigationfrequency'%3Astring%2C'locationfieldslocationtype'%3Astring%2C'DSRsourcetechnologyintroduction'%3Astring%2C'IIWMguide'%3Astring%2C'village'%3Astring%2C'IWMstaleseedbed'%3Astring%2C'ICarea'%3Anumber%2C'locationAltitude'%3Alocation%2C'SPinitialyear'%3Astring%2C'ICinitialyear'%3Astring%2C'RBPwidth'%3Astring%2C'RTtype'%3Astring%2C'RBPresidueretentionpreviouscrop'%3Astring%2C'CSDintroducedcrop'%3Astring%2C'DSRirrigation'%3Astring%2C'preDSRpractice'%3Astring%2C'LLLarea'%3Anumber%2C'manuallatitude'%3Anumber%2C'RCsourcetechnologyintroduction'%3Astring%2C'INMliming'%3Astring%2C'IIWMcrop'%3Astring%2C'RTmaincroppingsystem'%3Astring%2C'RCmaincrops'%3Astring%2C'ICmaincroppingsystem'%3Astring%2C'INMothernutrients'%3Astring%2C'UPTRmaincroppingsystem'%3Astring%2C'comment'%3Astring%2C'IIWMinitialyear'%3Astring%2C'farmarea'%3Anumber%2C'IIWMsourcetechnologyintroduction'%3Astring%2C'LLLmaincroppingsystem'%3Astring%2C'ZTmaincroppingsystem'%3Astring%2C'IWMUPTR'%3Astring%2C'IPHSinitialyear'%3Astring%2C'RCmaincroppingsystem'%3Astring%2C'LLLZT'%3Astring%2C'ZTinitialyear'%3Astring%2C'IWMZT'%3Astring%2C'CSDreplacedcrop'%3Astring%2C'RBPtype'%3Astring%2C'DSRirrigationscheduling'%3Astring%2C'locationAccuracy'%3Alocation%2C'adoptedtechnologies'%3Astring%2C'IPHSstoragetype'%3Astring%2C'surveyordetailssurveymethod'%3Astring%2C'IWMherbicides'%3Astring%2C'INMarea'%3Anumber%2C'RBParea'%3Anumber%2C'district'%3Astring%2C'IWMotherweedmanagement'%3Astring%2C'IPHSstoragecapacity'%3Anumber%2C'IWMbrownmanuring'%3Astring%2C'SubmissionDate'%3Adatetime%2C'LLLRT'%3Astring%2C'CFpreviouscropfallowperiod'%3Astring%2C'DSRarea'%3Anumber%2C'IWMRT'%3Astring%2C'INMguide'%3Astring%2C'farmerdetailsnamefarmer'%3Astring%2C'INMnutrients'%3Astring%2C'IWMsourcetechnologyintroduction'%3Astring%2C'LLLsourcetechnologyintroduction'%3Astring%2C'RTsourcetechnologyintroduction'%3Astring%2C'telephone'%3Anumber%2C'NCsourcetechnologyintroduction'%3Astring%2C'SPcrop'%3Astring%2C'location'%3Alocation%2C'percentagefarmarearented'%3Astring%2C'ZTarea'%3Anumber%2C'INMNPKincrease'%3Astring%2C'RBPmaincroppingsystem'%3Astring%2C'CFseasonnewcrop'%3Astring%2C'IIWMchangedirrigationtiming'%3Astring%2C'IWMherbicideapplicationfrequency'%3Astring%2C'CFinitialyear'%3Astring%2C'DSRmethod'%3Astring%2C'ICintercrop'%3Astring%2C'ICmaincrops'%3Astring%2C'CSDmaincroppingsystem'%3Astring%2C'IWMarea'%3Anumber%2C'UPTRarea'%3Anumber%2C'IWMDSR'%3Astring%2C'RTarea'%3Anumber%2C'NColdvariety'%3Astring%2C'RCarea'%3Anumber%2C'IPHSsourcetechnologyintroduction'%3Astring%2C'NCinitialyear'%3Astring%2C'DSRinitialyear'%3Astring%2C'INMsplitnitrogenapplication'%3Astring%2C'UPTRRT'%3Astring%2C'NCDSR'%3Astring%2C'CSDarea'%3Anumber%2C'RBPcrop'%3Astring%2C'RCinitialyear'%3Astring%2C'UPTRZT'%3Astring%2C'ZTcrop'%3Astring%2C'IWMinitialyear'%3Astring%2C'IIWMDSR'%3Astring%2C'ICsourcetechnologyintroduction'%3Astring%2C'SPseedtreatment'%3Astring%2C'preUPTRpractice'%3Astring%2C'ZTresidueretentionpreviouscrop'%3Astring%2C'INMbandedfertilizerapplication'%3Astring%2C'IIWMarea'%3Anumber%2C'RTcrop'%3Astring%2C'LLLlastyear'%3Astring%2C'NCZT'%3Astring%2C'CFsourcetechnologyintroduction'%3Astring%2C'ZTsourcetechnologyintroduction'%3Astring%2C'IWMRBP'%3Astring%2C'INMNPKdecrease'%3Astring%2C'RTinitialyear'%3Astring%2C'CSDinitialyear'%3Astring%2C'farmerdetailsfarmergender'%3Astring%2C'RTpriortillagepasses'%3Anumber%2C'NCnewvariety'%3Astring%2C'DSRmaincroppingsystem'%3Astring%2C'UPTRLLL'%3Astring%2C'NCcrop'%3Astring%2C'INMureagranulesplacement'%3Astring%2C'ZTpriortillagepasses'%3Anumber%2C'DSRZT'%3Astring%2C'DSRLLL'%3Astring%2C'RBPinitialyear'%3Astring%2C'manuallongitude'%3Anumber%2C'RTnumbertillages'%3Anumber%2C'NCRT'%3Astring%2C'UPTRtransplantingtype'%3Astring%2C'UPTRsourcetechnologyintroduction'%3Astring%2C'UPTRinitialyear'%3Astring%2C'IIWMUPTR'%3Astring%2C'RTresidueretentionpreviouscrop'%3Astring%2C'DSRRT'%3Astring%2C'INMinitialyear'%3Astring%2C'CSDsourcetechnologyintroduction'%3Astring%2C'INMsourcetechnologyintroduction'%3Astring%2C'IWMcrop'%3Astring%2C'INMcrop'%3Astring)

--
Post: opendatakit@googlegroups.com
Unsubscribe: opendatakit+unsubscribe@googlegroups.com
Options: http://groups.google.com/group/opendatakit?hl=en

--
Mitch Sundt
Software Engineer

University of Washington
mitchellsundt@gmail.com

David · March 31, 2011, 11:41pm

Mitch,

Thanks for the response! In case it makes a difference, I note that
even when I shorten the field names, the same error is still
returned. I tried shortening the names quite a bit, and this resulted
in an error with a slightly shorter reported URL.

How long might this take to fix in Aggregate?

Also, I've had a problem on slow internet connections in which ODK
Collect reports that there is failure in submitting records to
Aggregate (0.95 on appspot), but it actually turns out that they have
been at least partially transmitted. When one resubmits, this then
results in duplicates of the same record in Aggregate. Is there a fix
for this?

Best,
David

···

On Apr 1, 2:00 am, Mitch Sundt wrote: > Hi David, > > We are passing the sql=CREATE... parameter as an argument on the POST > request's URL, instead of in the body of the request, and that is making the > URL string too long. It looks like FusionTables supports passing the > parameter in the body of the POST request, which should eliminate this > limitation. > > I'll see what we can do to change this. > > Mitch > > On Thu, Mar 31, 2011 at 7:00 AM, David A. Raitzer wrote: > > > Dear all, > > > Any help that can be offered would be much appreciated. When I try to set up an ODK Aggregate (v.0.95) connection to Google Fusion tables, the error below is returned. I've tried shortening field names, changing survey IDs, and nothing seems to help. ODK can set up a connection to Google Spreadsheets, but no data are transmitted other than submission times. > > > What is the problem? The XML is attached for reference. > > > Thanks in advance. > > > Best, > > David > > > Uncaught exception from servlet > > java.net.MalformedURLException: Invalid URL:http://tables.googlelabs.com/api/query?sql=CREATE+TABLE+%27AdoptionSu... > > > at com.google.appengine.api.urlfetch.URLFetchServiceImpl.convertApplicationException(URLFetchServiceImpl.java:106) > > at com.google.appengine.api.urlfetch.URLFetchServiceImpl.fetch(URLFetchServiceImpl.java:41) > > at com.google.apphosting.utils.security.urlfetch.URLFetchServiceStreamHandler$Connection.fetchResponse(URLFetchServiceStreamHandler.java:418) > > > at com.google.apphosting.utils.security.urlfetch.URLFetchServiceStreamHandler$Connection.getInputStream(URLFetchServiceStreamHandler.java:297) > > at org.odk.aggregate.form.remoteserver.FusionTableOAuth.executeInsert(FusionTableOAuth.java:149) > > > at org.odk.aggregate.servlet.FusionTableServlet.doGet(FusionTableServlet.java:140) > > at javax.servlet.http.HttpServlet.service(HttpServlet.java:617) > > at javax.servlet.http.HttpServlet.service(HttpServlet.java:717) > > > at org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:511) > > at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1166) > > at com.google.apphosting.utils.servlet.ParseBlobUploadFilter.doFilter(ParseBlobUploadFilter.java:97) > > > at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1157) > > at com.google.apphosting.runtime.jetty.SaveSessionFilter.doFilter(SaveSessionFilter.java:35) > > at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1157) > > > at com.google.apphosting.utils.servlet.TransactionCleanupFilter.doFilter(TransactionCleanupFilter.java:43) > > at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1157) > > at org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:388) > > > at org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216) > > at org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:182) > > at org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:765) > > > at org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:418) > > at com.google.apphosting.runtime.jetty.AppVersionHandlerMap.handle(AppVersionHandlerMap.java:238) > > at org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152) > > > at org.mortbay.jetty.Server.handle(Server.java:326) > > at org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:542) > > at org.mortbay.jetty.HttpConnection$RequestHandler.headerComplete(HttpConnection.java:923) > > > at com.google.apphosting.runtime.jetty.RpcRequestParser.parseAvailable(RpcRequestParser.java:76) > > at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:404) > > at com.google.apphosting.runtime.jetty.JettyServletEngineAdapter.serviceRequest(JettyServletEngineAdapter.java:135) > > > at com.google.apphosting.runtime.JavaRuntime.handleRequest(JavaRuntime.java:261) > > at com.google.apphosting.base.RuntimePb$EvaluationRuntime$2.handleRequest(RuntimePb.java:9285) > > at com.google.net.rpc.impl.RpcUtil.runRpcInApplication(RpcUtil.java:437) > > > at com.google.net.rpc.impl.Server$RpcTask.runInContext(Server.java:573) > > at com.google.tracing.TraceContext$TraceContextRunnable$1.run(TraceContext.java:448) > > at com.google.tracing.TraceContext.runInContext(TraceContext.java:688) > > > at com.google.tracing.TraceContext$AbstractTraceContextCallback.runInInheritedContextNoUnref(TraceContext.java:326) > > at com.google.tracing.TraceContext$AbstractTraceContextCallback.runInInheritedContext(TraceContext.java:318) > > > Unexpected exception from servlet: java.net.MalformedURLException: Invalid URL: > > ... > > read more »

David · April 1, 2011, 1:20am

Mitch,

Here is one more observation. On your demo server (http://
opendatakit.appspot.com/forms?null), the form named "eIMCI by D-Tree"
has an even longer number of characters in the field titles (2575 to
be exact) than does my form (which has 2353). However, it can
successfully establish a connection to Fusion Tables. So, does that
mean that there could be another problem at hand? Thanks again.

Best,
David

···

On Apr 1, 7:41 am, David wrote: > Mitch, > > Thanks for the response! In case it makes a difference, I note that > even when I shorten the field names, the same error is still > returned. I tried shortening the names quite a bit, and this resulted > in an error with a slightly shorter reported URL. > > How long might this take to fix in Aggregate? > > Also, I've had a problem on slow internet connections in which ODK > Collect reports that there is failure in submitting records to > Aggregate (0.95 on appspot), but it actually turns out that they have > been at least partially transmitted. When one resubmits, this then > results in duplicates of the same record in Aggregate. Is there a fix > for this? > > Best, > David > > On Apr 1, 2:00 am, Mitch Sundt wrote: > > > Hi David, > > > We are passing the sql=CREATE... parameter as an argument on the POST > > request's URL, instead of in the body of the request, and that is making the > > URL string too long. It looks like FusionTables supports passing the > > parameter in the body of the POST request, which should eliminate this > > limitation. > > > I'll see what we can do to change this. > > > Mitch > > > On Thu, Mar 31, 2011 at 7:00 AM, David A. Raitzer wrote: > > > > Dear all, > > > > Any help that can be offered would be much appreciated. When I try to set up an ODK Aggregate (v.0.95) connection to Google Fusion tables, the error below is returned. I've tried shortening field names, changing survey IDs, and nothing seems to help. ODK can set up a connection to Google Spreadsheets, but no data are transmitted other than submission times. > > > > What is the problem? The XML is attached for reference. > > > > Thanks in advance. > > > > Best, > > > David > > > > Uncaught exception from servlet > > > java.net.MalformedURLException: Invalid URL:http://tables.googlelabs.com/api/query?sql=CREATE+TABLE+%27AdoptionSu... > > > > at com.google.appengine.api.urlfetch.URLFetchServiceImpl.convertApplicationException(URLFetchServiceImpl.java:106) > > > at com.google.appengine.api.urlfetch.URLFetchServiceImpl.fetch(URLFetchServiceImpl.java:41) > > > at com.google.apphosting.utils.security.urlfetch.URLFetchServiceStreamHandler$Connection.fetchResponse(URLFetchServiceStreamHandler.java:418) > > > > at com.google.apphosting.utils.security.urlfetch.URLFetchServiceStreamHandler$Connection.getInputStream(URLFetchServiceStreamHandler.java:297) > > > at org.odk.aggregate.form.remoteserver.FusionTableOAuth.executeInsert(FusionTableOAuth.java:149) > > > > at org.odk.aggregate.servlet.FusionTableServlet.doGet(FusionTableServlet.java:140) > > > at javax.servlet.http.HttpServlet.service(HttpServlet.java:617) > > > at javax.servlet.http.HttpServlet.service(HttpServlet.java:717) > > > > at org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:511) > > > at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1166) > > > at com.google.apphosting.utils.servlet.ParseBlobUploadFilter.doFilter(ParseBlobUploadFilter.java:97) > > > > at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1157) > > > at com.google.apphosting.runtime.jetty.SaveSessionFilter.doFilter(SaveSessionFilter.java:35) > > > at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1157) > > > > at com.google.apphosting.utils.servlet.TransactionCleanupFilter.doFilter(TransactionCleanupFilter.java:43) > > > at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1157) > > > at org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:388) > > > > at org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216) > > > at org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:182) > > > at org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:765) > > > > at org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:418) > > > at com.google.apphosting.runtime.jetty.AppVersionHandlerMap.handle(AppVersionHandlerMap.java:238) > > > at org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152) > > > > at org.mortbay.jetty.Server.handle(Server.java:326) > > > at org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:542) > > > at org.mortbay.jetty.HttpConnection$RequestHandler.headerComplete(HttpConnection.java:923) > > > > at com.google.apphosting.runtime.jetty.RpcRequestParser.parseAvailable(RpcRequestParser.java:76) > > > at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:404) > > > at com.google.apphosting.runtime.jetty.JettyServletEngineAdapter.serviceRequest(JettyServletEngineAdapter.java:135) > > > > at com.google.apphosting.runtime.JavaRuntime.handleRequest(JavaRuntime.java:261) > > > at com.google.apphosting.base.RuntimePb$EvaluationRuntime$2.handleRequest(RuntimePb.java:9285) > > > at com.google.net.rpc.impl.RpcUtil.runRpcInApplication(RpcUtil.java:437) > > > > at com.google.net.rpc.impl.Server$RpcTask.runInContext(Server.java:573) > > > at com.google.tracing.TraceContext$TraceContextRunnable$1.run(TraceContext.java:448) > > > at com.google.tracing.TraceContext.runInContext(TraceContext.java:688) > > > > at com.google.tracing.TraceContext$AbstractTraceContextCallback.runInInheritedContextNoUnref(TraceContext.java:326) > > > at com.google.tracing.TraceContext$AbstractTraceContextCallback.runInInheritedContext(TraceContext.java:318) > > > > Unexpected exception from servlet: java.net.MalformedURLException: Invalid URL: > > > ... > > > read more »

Mitch_Sundt · April 1, 2011, 7:56pm

Thanks for the additional sleuthing!

I am not certain why yours did not work and eIMCI did, as they are both
large surveys.

I have confirmed through testing that the initial issue is that the URL is
too long
and that by moving the 'sql' argument into the body, it successfully
creates the table. There is a secondary issue with the
location-fields/location
field in your form which appears to not go through properly when submitting
data. I suspect this is because you don't gather that field data as a
geopoint, yet
you have the type defined as a geopoint.

The final fix requires that I write a library. Working on that now.

The submission of duplicates is a long-standing issue. The OpenRosa
community
is defining a 'metadata' section for compliant forms that contains an
'instanceId' field
that would be a unique id that servers could use to identify repeat
submissions. The
support for this is not yet available.
https://bitbucket.org/javarosa/javarosa/wiki/OpenRosaMetaDataSchema

Without that, there is only the manual comparisons of data rows and the
manual deletions of
the duplicates. Recording start and end times and the imei of the
submitting phone
can be used to simplify the identification of duplicate entries.

Mitch

···

On Thu, Mar 31, 2011 at 6:20 PM, David wrote:

Mitch,

Here is one more observation. On your demo server (http://
opendatakit.appspot.com/forms?null), the form named "eIMCI by D-Tree"
has an even longer number of characters in the field titles (2575 to
be exact) than does my form (which has 2353). However, it can
successfully establish a connection to Fusion Tables. So, does that
mean that there could be another problem at hand? Thanks again.

Best,
David

On Apr 1, 7:41 am, David david.rait...@gmail.com wrote:

Mitch,

Thanks for the response! In case it makes a difference, I note that
even when I shorten the field names, the same error is still
returned. I tried shortening the names quite a bit, and this resulted
in an error with a slightly shorter reported URL.

How long might this take to fix in Aggregate?

Also, I've had a problem on slow internet connections in which ODK
Collect reports that there is failure in submitting records to
Aggregate (0.95 on appspot), but it actually turns out that they have
been at least partially transmitted. When one resubmits, this then
results in duplicates of the same record in Aggregate. Is there a fix
for this?

Best,
David

On Apr 1, 2:00 am, Mitch Sundt msu...@cs.washington.edu wrote:

Hi David,

We are passing the sql=CREATE... parameter as an argument on the POST
request's URL, instead of in the body of the request, and that is
making the
URL string too long. It looks like FusionTables supports passing the
parameter in the body of the POST request, which should eliminate this
limitation.

I'll see what we can do to change this.

Mitch

On Thu, Mar 31, 2011 at 7:00 AM, David A. Raitzer david.rait...@gmail.comwrote:

Dear all,

Any help that can be offered would be much appreciated. When I try
to set up an ODK Aggregate (v.0.95) connection to Google Fusion tables, the
error below is returned. I've tried shortening field names, changing survey
IDs, and nothing seems to help. ODK can set up a connection to Google
Spreadsheets, but no data are transmitted other than submission times.

What is the problem? The XML is attached for reference.

Thanks in advance.

Best,
David

Uncaught exception from servlet
java.net.MalformedURLException: Invalid URL:
http://tables.googlelabs.com/api/query?sql=CREATE+TABLE+'AdoptionSu...

at
com.google.appengine.api.urlfetch.URLFetchServiceImpl.convertApplicationException(URLFetchServiceImpl.java:106)
at
com.google.appengine.api.urlfetch.URLFetchServiceImpl.fetch(URLFetchServiceImpl.java:41)
at
com.google.apphosting.utils.security.urlfetch.URLFetchServiceStreamHandler$Connection.fetchResponse(URLFetchServiceStreamHandler.java:418)

at
com.google.apphosting.utils.security.urlfetch.URLFetchServiceStreamHandler$Connection.getInputStream(URLFetchServiceStreamHandler.java:297)
at
org.odk.aggregate.form.remoteserver.FusionTableOAuth.executeInsert(FusionTableOAuth.java:149)

at
org.odk.aggregate.servlet.FusionTableServlet.doGet(FusionTableServlet.java:140)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:617)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:717)

at
org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:511)
at
org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1166)
at
com.google.apphosting.utils.servlet.ParseBlobUploadFilter.doFilter(ParseBlobUploadFilter.java:97)

at
org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1157)
at
com.google.apphosting.runtime.jetty.SaveSessionFilter.doFilter(SaveSessionFilter.java:35)
at
org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1157)

at
com.google.apphosting.utils.servlet.TransactionCleanupFilter.doFilter(TransactionCleanupFilter.java:43)
at
org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1157)
at
org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:388)

at
org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216)
at
org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:182)
at
org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:765)

at
org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:418)
at
com.google.apphosting.runtime.jetty.AppVersionHandlerMap.handle(AppVersionHandlerMap.java:238)
at
org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152)

at org.mortbay.jetty.Server.handle(Server.java:326)
at
org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:542)
at
org.mortbay.jetty.HttpConnection$RequestHandler.headerComplete(HttpConnection.java:923)

at
com.google.apphosting.runtime.jetty.RpcRequestParser.parseAvailable(RpcRequestParser.java:76)
at
org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:404)
at
com.google.apphosting.runtime.jetty.JettyServletEngineAdapter.serviceRequest(JettyServletEngineAdapter.java:135)

at
com.google.apphosting.runtime.JavaRuntime.handleRequest(JavaRuntime.java:261)
at
com.google.apphosting.base.RuntimePb$EvaluationRuntime$2.handleRequest(RuntimePb.java:9285)
at
com.google.net.rpc.impl.RpcUtil.runRpcInApplication(RpcUtil.java:437)

at
com.google.net.rpc.impl.Server$RpcTask.runInContext(Server.java:573)
at
com.google.tracing.TraceContext$TraceContextRunnable$1.run(TraceContext.java:448)
at
com.google.tracing.TraceContext.runInContext(TraceContext.java:688)

at
com.google.tracing.TraceContext$AbstractTraceContextCallback.runInInheritedContextNoUnref(TraceContext.java:326)
at
com.google.tracing.TraceContext$AbstractTraceContextCallback.runInInheritedContext(TraceContext.java:318)

Unexpected exception from servlet: java.net.MalformedURLException:
Invalid URL:

...

read more »

--
Post: opendatakit@googlegroups.com
Unsubscribe: opendatakit+unsubscribe@googlegroups.com
Options: http://groups.google.com/group/opendatakit?hl=en

--
Mitch Sundt
Software Engineer

University of Washington
mitchellsundt@gmail.com

David · April 1, 2011, 8:55pm

Thanks again for the help. How should the location field be corrected
in the XML?

Best,
David

···

On Apr 2, 3:56 am, Mitch Sundt wrote: > Thanks for the additional sleuthing! > > I am not certain why yours did not work and eIMCI did, as they are both > large surveys. > > I have confirmed through testing that the initial issue is that the URL is > too long > and that by moving the 'sql' argument into the body, it successfully > creates the table. There is a secondary issue with the > location-fields/location > field in your form which appears to not go through properly when submitting > data. I suspect this is because you don't gather that field data as a > geopoint, yet > you have the type defined as a geopoint. > > The final fix requires that I write a library. Working on that now. > > The submission of duplicates is a long-standing issue. The OpenRosa > community > is defining a 'metadata' section for compliant forms that contains an > 'instanceId' field > that would be a unique id that servers could use to identify repeat > submissions. The > support for this is not yet available.https://bitbucket.org/javarosa/javarosa/wiki/OpenRosaMetaDataSchema > > Without that, there is only the manual comparisons of data rows and the > manual deletions of > the duplicates. Recording start and end times and the imei of the > submitting phone > can be used to simplify the identification of duplicate entries. > > Mitch > > > > > > On Thu, Mar 31, 2011 at 6:20 PM, David wrote: > > Mitch, > > > Here is one more observation. On your demo server (http:// > > opendatakit.appspot.com/forms?null), the form named "eIMCI by D-Tree" > > has an even longer number of characters in the field titles (2575 to > > be exact) than does my form (which has 2353). However, it can > > successfully establish a connection to Fusion Tables. So, does that > > mean that there could be another problem at hand? Thanks again. > > > Best, > > David > > > On Apr 1, 7:41 am, David wrote: > > > Mitch, > > > > Thanks for the response! In case it makes a difference, I note that > > > even when I shorten the field names, the same error is still > > > returned. I tried shortening the names quite a bit, and this resulted > > > in an error with a slightly shorter reported URL. > > > > How long might this take to fix in Aggregate? > > > > Also, I've had a problem on slow internet connections in which ODK > > > Collect reports that there is failure in submitting records to > > > Aggregate (0.95 on appspot), but it actually turns out that they have > > > been at least partially transmitted. When one resubmits, this then > > > results in duplicates of the same record in Aggregate. Is there a fix > > > for this? > > > > Best, > > > David > > > > On Apr 1, 2:00 am, Mitch Sundt wrote: > > > > > Hi David, > > > > > We are passing the sql=CREATE... parameter as an argument on the POST > > > > request's URL, instead of in the body of the request, and that is > > making the > > > > URL string too long. It looks like FusionTables supports passing the > > > > parameter in the body of the POST request, which should eliminate this > > > > limitation. > > > > > I'll see what we can do to change this. > > > > > Mitch > > > > > On Thu, Mar 31, 2011 at 7:00 AM, David A. Raitzer wrote: > > > > > > Dear all, > > > > > > Any help that can be offered would be much appreciated. When I try > > to set up an ODK Aggregate (v.0.95) connection to Google Fusion tables, the > > error below is returned. I've tried shortening field names, changing survey > > IDs, and nothing seems to help. ODK can set up a connection to Google > > Spreadsheets, but no data are transmitted other than submission times. > > > > > > What is the problem? The XML is attached for reference. > > > > > > Thanks in advance. > > > > > > Best, > > > > > David > > > > > > Uncaught exception from servlet > > > > > java.net.MalformedURLException: Invalid URL: > >http://tables.googlelabs.com/api/query?sql=CREATE+TABLE+%27AdoptionSu... > > > > > > at > > com.google.appengine.api.urlfetch.URLFetchServiceImpl.convertApplicationExc eption(URLFetchServiceImpl.java:106) > > > > > at > > com.google.appengine.api.urlfetch.URLFetchServiceImpl.fetch(URLFetchService Impl.java:41) > > > > > at > > com.google.apphosting.utils.security.urlfetch.URLFetchServiceStreamHandler$ Connection.fetchResponse(URLFetchServiceStreamHandler.java:418) > > > > > > at > > com.google.apphosting.utils.security.urlfetch.URLFetchServiceStreamHandler$ Connection.getInputStream(URLFetchServiceStreamHandler.java:297) > > > > > at > > org.odk.aggregate.form.remoteserver.FusionTableOAuth.executeInsert(FusionTa bleOAuth.java:149) > > > > > > at > > org.odk.aggregate.servlet.FusionTableServlet.doGet(FusionTableServlet.java: 140) > > > > > at javax.servlet.http.HttpServlet.service(HttpServlet.java:617) > > > > > at javax.servlet.http.HttpServlet.service(HttpServlet.java:717) > > > > > > at > > org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:511) > > > > > at > > org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandle r.java:1166) > > > > > at > > com.google.apphosting.utils.servlet.ParseBlobUploadFilter.doFilter(ParseBlo bUploadFilter.java:97) > > > > > > at > > org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandle r.java:1157) > > > > > at > > com.google.apphosting.runtime.jetty.SaveSessionFilter.doFilter(SaveSessionF ilter.java:35) > > > > > at > > org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandle r.java:1157) > > > > > > at > > com.google.apphosting.utils.servlet.TransactionCleanupFilter.doFilter(Trans actionCleanupFilter.java:43) > > > > > at > > org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandle r.java:1157) > > > > > at > > org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:388) > > > > > > at > > org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216) > > > > > at > > org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:182) > > > > > at > > org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:765) > > > > > > at > > org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:418) > > > > > at > > com.google.apphosting.runtime.jetty.AppVersionHandlerMap.handle(AppVersionH andlerMap.java:238) > > > > > at > > org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152) > > > > > > at org.mortbay.jetty.Server.handle(Server.java:326) > > > > > at > > org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:542) > > > > > at > > org.mortbay.jetty.HttpConnection$RequestHandler.headerComplete(HttpConnecti on.java:923) > > > > > > at > > com.google.apphosting.runtime.jetty.RpcRequestParser.parseAvailable(RpcRequ estParser.java:76) > > > > > at > > org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:404) > > > > > at > > com.google.apphosting.runtime.jetty.JettyServletEngineAdapter.serviceReques t(JettyServletEngineAdapter.java:135) > > > > > > at > > com.google.apphosting.runtime.JavaRuntime.handleRequest(JavaRuntime.java:26 1) > > > > > at > > com.google.apphosting.base.RuntimePb$EvaluationRuntime$2.handleRequest(Runt imePb.java:9285) > > > > > at > > com.google.net.rpc.impl.RpcUtil.runRpcInApplication(RpcUtil.java:437) > > > > > > at > > com.google.net.rpc.impl.Server$RpcTask.runInContext(Server.java:573) > > > > > at > > com.google.tracing.TraceContext$TraceContextRunnable$1.run(TraceContext.jav a:448) > > > > > at > > com.google.tracing.TraceContext.runInContext(TraceContext.java:688) > > > > > > at > > com.google.tracing.TraceContext$AbstractTraceContextCallback.runInInherited ContextNoUnref(TraceContext.java:326) > > > > > at > > com.google.tracing.TraceContext$AbstractTraceContextCallback.runInInherited Context(TraceContext.java:318) > > > > > > Unexpected exception from servlet: java.net.MalformedURLException: > > Invalid URL: > > > > > ... > > > > > read more » > > > -- > > Post: opendatakit@googlegroups.com > > Unsubscribe: opendatakit+unsubscribe@googlegroups.com > > Options:http://groups.google.com/group/opendatakit?hl=en > > -- > Mitch Sundt > Software Engineerhttp://www.OpenDataKit.org > University of Washington > mitchellsu...@gmail.com

Mitch_Sundt · April 1, 2011, 9:07pm

Short answer: I Don't know.
You should verify that you can upload submissions and view them OK on
Aggregate.

Location should have a type of 'geopoint' e.g.,

···

On Fri, Apr 1, 2011 at 1:55 PM, David wrote:

Thanks again for the help. How should the location field be corrected
in the XML?

Best,
David

On Apr 2, 3:56 am, Mitch Sundt msu...@cs.washington.edu wrote:

Thanks for the additional sleuthing!

I am not certain why yours did not work and eIMCI did, as they are both
large surveys.

I have confirmed through testing that the initial issue is that the URL
is
too long
and that by moving the 'sql' argument into the body, it successfully
creates the table. There is a secondary issue with the
location-fields/location
field in your form which appears to not go through properly when
submitting
data. I suspect this is because you don't gather that field data as a
geopoint, yet
you have the type defined as a geopoint.

The final fix requires that I write a library. Working on that now.

The submission of duplicates is a long-standing issue. The OpenRosa
community
is defining a 'metadata' section for compliant forms that contains an
'instanceId' field
that would be a unique id that servers could use to identify repeat
submissions. The
support for this is not yet available.
https://bitbucket.org/javarosa/javarosa/wiki/OpenRosaMetaDataSchema

Without that, there is only the manual comparisons of data rows and the
manual deletions of
the duplicates. Recording start and end times and the imei of the
submitting phone
can be used to simplify the identification of duplicate entries.

Mitch

On Thu, Mar 31, 2011 at 6:20 PM, David david.rait...@gmail.com wrote:

Mitch,

Here is one more observation. On your demo server (http://
opendatakit.appspot.com/forms?null), the form named "eIMCI by D-Tree"
has an even longer number of characters in the field titles (2575 to
be exact) than does my form (which has 2353). However, it can
successfully establish a connection to Fusion Tables. So, does that
mean that there could be another problem at hand? Thanks again.

Best,
David

On Apr 1, 7:41 am, David david.rait...@gmail.com wrote:

Mitch,

Thanks for the response! In case it makes a difference, I note that
even when I shorten the field names, the same error is still
returned. I tried shortening the names quite a bit, and this
resulted
in an error with a slightly shorter reported URL.

How long might this take to fix in Aggregate?

Also, I've had a problem on slow internet connections in which ODK
Collect reports that there is failure in submitting records to
Aggregate (0.95 on appspot), but it actually turns out that they have
been at least partially transmitted. When one resubmits, this then
results in duplicates of the same record in Aggregate. Is there a
fix
for this?

Best,
David

On Apr 1, 2:00 am, Mitch Sundt msu...@cs.washington.edu wrote:

Hi David,

We are passing the sql=CREATE... parameter as an argument on the
POST
request's URL, instead of in the body of the request, and that is
making the
URL string too long. It looks like FusionTables supports passing
the
parameter in the body of the POST request, which should eliminate
this
limitation.

I'll see what we can do to change this.

Mitch

On Thu, Mar 31, 2011 at 7:00 AM, David A. Raitzer david.rait...@gmail.comwrote:

Dear all,

Any help that can be offered would be much appreciated. When I
try
to set up an ODK Aggregate (v.0.95) connection to Google Fusion tables,
the
error below is returned. I've tried shortening field names, changing
survey
IDs, and nothing seems to help. ODK can set up a connection to Google
Spreadsheets, but no data are transmitted other than submission times.

What is the problem? The XML is attached for reference.

Thanks in advance.

Best,
David

Uncaught exception from servlet
java.net.MalformedURLException: Invalid URL:
http://tables.googlelabs.com/api/query?sql=CREATE+TABLE+'AdoptionSu.
..

at

com.google.appengine.api.urlfetch.URLFetchServiceImpl.convertApplicationExc
eption(URLFetchServiceImpl.java:106)

at

com.google.appengine.api.urlfetch.URLFetchServiceImpl.fetch(URLFetchService
Impl.java:41)

at

com.google.apphosting.utils.security.urlfetch.URLFetchServiceStreamHandler$
Connection.fetchResponse(URLFetchServiceStreamHandler.java:418)

at

com.google.apphosting.utils.security.urlfetch.URLFetchServiceStreamHandler$
Connection.getInputStream(URLFetchServiceStreamHandler.java:297)

at

org.odk.aggregate.form.remoteserver.FusionTableOAuth.executeInsert(FusionTa
bleOAuth.java:149)

at

org.odk.aggregate.servlet.FusionTableServlet.doGet(FusionTableServlet.java:
140)

at
javax.servlet.http.HttpServlet.service(HttpServlet.java:617)
at
javax.servlet.http.HttpServlet.service(HttpServlet.java:717)

at
org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:511)
at

org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandle
r.java:1166)

at

com.google.apphosting.utils.servlet.ParseBlobUploadFilter.doFilter(ParseBlo
bUploadFilter.java:97)

at

org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandle
r.java:1157)

at

com.google.apphosting.runtime.jetty.SaveSessionFilter.doFilter(SaveSessionF
ilter.java:35)

at

org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandle
r.java:1157)

at

com.google.apphosting.utils.servlet.TransactionCleanupFilter.doFilter(Trans
actionCleanupFilter.java:43)

at

org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandle
r.java:1157)

at

org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:388)

at

org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216)

at

org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:182)

at

org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:765)

at
org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:418)
at

com.google.apphosting.runtime.jetty.AppVersionHandlerMap.handle(AppVersionH
andlerMap.java:238)

at

org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152)

at org.mortbay.jetty.Server.handle(Server.java:326)
at
org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:542)
at

org.mortbay.jetty.HttpConnection$RequestHandler.headerComplete(HttpConnecti
on.java:923)

at

com.google.apphosting.runtime.jetty.RpcRequestParser.parseAvailable(RpcRequ
estParser.java:76)

at
org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:404)
at

com.google.apphosting.runtime.jetty.JettyServletEngineAdapter.serviceReques
t(JettyServletEngineAdapter.java:135)

at

com.google.apphosting.runtime.JavaRuntime.handleRequest(JavaRuntime.java:26
1)

at

com.google.apphosting.base.RuntimePb$EvaluationRuntime$2.handleRequest(Runt
imePb.java:9285)

at
com.google.net.rpc.impl.RpcUtil.runRpcInApplication(RpcUtil.java:437)

at
com.google.net.rpc.impl.Server$RpcTask.runInContext(Server.java:573)
at

com.google.tracing.TraceContext$TraceContextRunnable$1.run(TraceContext.jav
a:448)

at
com.google.tracing.TraceContext.runInContext(TraceContext.java:688)

at

com.google.tracing.TraceContext$AbstractTraceContextCallback.runInInherited
ContextNoUnref(TraceContext.java:326)

at

com.google.tracing.TraceContext$AbstractTraceContextCallback.runInInherited
Context(TraceContext.java:318)

Unexpected exception from servlet:
java.net.MalformedURLException:
Invalid URL:

...

read more »

--
Post: opendatakit@googlegroups.com
Unsubscribe: opendatakit+unsubscribe@googlegroups.com
Options:http://groups.google.com/group/opendatakit?hl=en

--
Mitch Sundt
Software Engineerhttp://www.OpenDataKit.org
University of Washington
mitchellsu...@gmail.com

--
Post: opendatakit@googlegroups.com
Unsubscribe: opendatakit+unsubscribe@googlegroups.com
Options: http://groups.google.com/group/opendatakit?hl=en

--
Mitch Sundt
Software Engineer
http://www.OpenDataKit.org
University of Washington
mitchellsundt@gmail.com

David · April 2, 2011, 2:49am

Mitch,

Thanks. The location type is geopoint, as indicated below, and I can
see the uploaded GPS submissions in Aggregate. Is there some other
problem then?

It might also be noted that a Google spreadsheet can be established,
but most of the data do not actually transfer. Is this also due to
the location issue?

On a side note, the following code works for generating a unique user
id in Collect: Perhaps it might be
possible in a future Aggregate release to automatically delete records
with duplicate IDs?

I look forward to the Aggregate version with the SQL argument in the
body.

The help is appreciated.

Best,
David

···

On Apr 2, 5:07 am, Mitch Sundt wrote: > Short answer: I Don't know. > You should verify that you can upload submissions and view them OK on > Aggregate. > > Location should have a type of 'geopoint' e.g., > > > > > > > > On Fri, Apr 1, 2011 at 1:55 PM, David wrote: > > Thanks again for the help. How should the location field be corrected > > in the XML? > > > Best, > > David > > > On Apr 2, 3:56 am, Mitch Sundt wrote: > > > Thanks for the additional sleuthing! > > > > I am not certain why yours did not work and eIMCI did, as they are both > > > large surveys. > > > > I have confirmed through testing that the initial issue is that the URL > > is > > > too long > > > and that by moving the 'sql' argument into the body, it successfully > > > creates the table. There is a secondary issue with the > > > location-fields/location > > > field in your form which appears to not go through properly when > > submitting > > > data. I suspect this is because you don't gather that field data as a > > > geopoint, yet > > > you have the type defined as a geopoint. > > > > The final fix requires that I write a library. Working on that now. > > > > The submission of duplicates is a long-standing issue. The OpenRosa > > > community > > > is defining a 'metadata' section for compliant forms that contains an > > > 'instanceId' field > > > that would be a unique id that servers could use to identify repeat > > > submissions. The > > > support for this is not yet available. > >https://bitbucket.org/javarosa/javarosa/wiki/OpenRosaMetaDataSchema > > > > Without that, there is only the manual comparisons of data rows and the > > > manual deletions of > > > the duplicates. Recording start and end times and the imei of the > > > submitting phone > > > can be used to simplify the identification of duplicate entries. > > > > Mitch > > > > On Thu, Mar 31, 2011 at 6:20 PM, David wrote: > > > > Mitch, > > > > > Here is one more observation. On your demo server (http:// > > > > opendatakit.appspot.com/forms?null), the form named "eIMCI by D-Tree" > > > > has an even longer number of characters in the field titles (2575 to > > > > be exact) than does my form (which has 2353). However, it can > > > > successfully establish a connection to Fusion Tables. So, does that > > > > mean that there could be another problem at hand? Thanks again. > > > > > Best, > > > > David > > > > > On Apr 1, 7:41 am, David wrote: > > > > > Mitch, > > > > > > Thanks for the response! In case it makes a difference, I note that > > > > > even when I shorten the field names, the same error is still > > > > > returned. I tried shortening the names quite a bit, and this > > resulted > > > > > in an error with a slightly shorter reported URL. > > > > > > How long might this take to fix in Aggregate? > > > > > > Also, I've had a problem on slow internet connections in which ODK > > > > > Collect reports that there is failure in submitting records to > > > > > Aggregate (0.95 on appspot), but it actually turns out that they have > > > > > been at least partially transmitted. When one resubmits, this then > > > > > results in duplicates of the same record in Aggregate. Is there a > > fix > > > > > for this? > > > > > > Best, > > > > > David > > > > > > On Apr 1, 2:00 am, Mitch Sundt wrote: > > > > > > > Hi David, > > > > > > > We are passing the sql=CREATE... parameter as an argument on the > > POST > > > > > > request's URL, instead of in the body of the request, and that is > > > > making the > > > > > > URL string too long. It looks like FusionTables supports passing > > the > > > > > > parameter in the body of the POST request, which should eliminate > > this > > > > > > limitation. > > > > > > > I'll see what we can do to change this. > > > > > > > Mitch > > > > > > > On Thu, Mar 31, 2011 at 7:00 AM, David A. Raitzer wrote: > > > > > > > > Dear all, > > > > > > > > Any help that can be offered would be much appreciated. When I > > try > > > > to set up an ODK Aggregate (v.0.95) connection to Google Fusion tables, > > the > > > > error below is returned. I've tried shortening field names, changing > > survey > > > > IDs, and nothing seems to help. ODK can set up a connection to Google > > > > Spreadsheets, but no data are transmitted other than submission times. > > > > > > > > What is the problem? The XML is attached for reference. > > > > > > > > Thanks in advance. > > > > > > > > Best, > > > > > > > David > > > > > > > > Uncaught exception from servlet > > > > > > > java.net.MalformedURLException: Invalid URL: > > > >http://tables.googlelabs.com/api/query?sql=CREATE+TABLE+%27AdoptionSu. > > .. > > > > > > > > at > > > com.google.appengine.api.urlfetch.URLFetchServiceImpl.convertApplicationExc > > eption(URLFetchServiceImpl.java:106) > > > > > > > at > > > com.google.appengine.api.urlfetch.URLFetchServiceImpl.fetch(URLFetchService > > Impl.java:41) > > > > > > > at > > > com.google.apphosting.utils.security.urlfetch.URLFetchServiceStreamHandler$ > > Connection.fetchResponse(URLFetchServiceStreamHandler.java:418) > > > > > > > > at > > > com.google.apphosting.utils.security.urlfetch.URLFetchServiceStreamHandler$ > > Connection.getInputStream(URLFetchServiceStreamHandler.java:297) > > > > > > > at > > > org.odk.aggregate.form.remoteserver.FusionTableOAuth.executeInsert(FusionTa > > bleOAuth.java:149) > > > > > > > > at > > > org.odk.aggregate.servlet.FusionTableServlet.doGet(FusionTableServlet.java: > > 140) > > > > > > > at > > javax.servlet.http.HttpServlet.service(HttpServlet.java:617) > > > > > > > at > > javax.servlet.http.HttpServlet.service(HttpServlet.java:717) > > > > > > > > at > > > > org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:511) > > > > > > > at > > > org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandle > > r.java:1166) > > > > > > > at > > > com.google.apphosting.utils.servlet.ParseBlobUploadFilter.doFilter(ParseBlo > > bUploadFilter.java:97) > > > > > > > > at > > > org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandle > > r.java:1157) > > > > > > > at > > > com.google.apphosting.runtime.jetty.SaveSessionFilter.doFilter(SaveSessionF > > ilter.java:35) > > > > > > > at > > > org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandle > > r.java:1157) > > > > > > > > at > > > com.google.apphosting.utils.servlet.TransactionCleanupFilter.doFilter(Trans > > actionCleanupFilter.java:43) > > > > > > > at > > > org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandle > > r.java:1157) > > > > > > > at > > > org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:388) > > > > > > > > at > > > org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216) > > > > > > > at > > > org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:182) > > > > > > > at > > > org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:765) > > > > > > > > at > > > > org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:418) > > > > > > > at > > > com.google.apphosting.runtime.jetty.AppVersionHandlerMap.handle(AppVersionH > > andlerMap.java:238) > > > > > > > at > > > org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152) > > > > > > > > at org.mortbay.jetty.Server.handle(Server.java:326) > > > > > > > at > > > > org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:542) > > > > > > > at > > > org.mortbay.jetty.HttpConnection$RequestHandler.headerComplete(HttpConnecti > > on.java:923) > > > > > > > > at > > > com.google.apphosting.runtime.jetty.RpcRequestParser.parseAvailable(RpcRequ > > estParser.java:76) > > > > > > > at > > > > org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:404) > > > > > > > at > > > com.google.apphosting.runtime.jetty.JettyServletEngineAdapter.serviceReques > > t(JettyServletEngineAdapter.java:135) > > > > > > > > at > > > com.google.apphosting.runtime.JavaRuntime.handleRequest(JavaRuntime.java:26 > > 1) > > > > > > > at > > > com.google.apphosting.base.RuntimePb$EvaluationRuntime$2.handleRequest(Runt > > imePb.java:9285) > > > > > > > at > > > > com.google.net.rpc.impl.RpcUtil.runRpcInApplication(RpcUtil.java:437) > > > > > > > > at > > > > com.google.net.rpc.impl.Server$RpcTask.runInContext(Server.java:573) > > > > > > > at > > > com.google.tracing.TraceContext$TraceContextRunnable$1.run(TraceContext.jav > > a:448) > > > > > > > at > > > > com.google.tracing.TraceContext.runInContext(TraceContext.java:688) > > > > > > > > at > > > com.google.tracing.TraceContext$AbstractTraceContextCallback.runInInherited > > ContextNoUnref(TraceContext.java:326) > > > > > > > at > > > com.google.tracing.TraceContext$AbstractTraceContextCallback.runInInherited > > Context(TraceContext.java:318) > > > > > > > > Unexpected exception from servlet: > > java.net.MalformedURLException: > > > > Invalid URL: > > > > > > > ... > > > > > > > read more » > > > > > -- > > > > Post: opendatakit@googlegroups.com > > > > Unsubscribe: opendatakit+unsubscribe@googlegroups.com > > > > Options:http://groups.google.com/group/opendatakit?hl=en > > > > -- > > > Mitch Sundt > > > Software Engineerhttp://www.OpenDataKit.org > > > University of Washington > > > mitchellsu...@gmail.com > > > -- > > Post: opendatakit@googlegroups.com > > Unsubscribe: opendatakit+unsubscribe@googlegroups.com > > Options:http://groups.google.com/group/opendatakit?hl=en > > -- > Mitch Sundt > Software Engineerhttp://www.OpenDataKit.org > University of Washington > mitchellsu...@gmail.com- Hide quoted text - > > - Show quoted text -

W_Brunette · April 2, 2011, 7:37am

David,

I believe we have fixes for the problems you listed with the 0.9.x
code base. The solution is currently available in the Aggregate code
repository at the tip of the default branch. Unfortunately the library
needed to allow a post to FusionTable instead of get request works
fine when running in the cloud but does not work when running locally.

We need to figure out how we are going to resolve this before pushing
a release. If you are in a hurry and want to run it on Google App
Engine in the cloud and are somewhat technically savy you can follow
Aggregate's advanced instructions and setup an eclipse project and
push the code yourself. If you aren't in a hurry and can wait until
early next week we should have a solution published.

Regards,
Waylon

···

On Fri, Apr 1, 2011 at 7:49 PM, David wrote: > Mitch, > > Thanks. The location type is geopoint, as indicated below, and I can > see the uploaded GPS submissions in Aggregate. Is there some other > problem then? > > It might also be noted that a Google spreadsheet can be established, > but most of the data do not actually transfer. Is this also due to > the location issue? > > On a side note, the following code works for generating a unique user > id in Collect: jr:preload="uid" jr:preloadParams="general" /> Perhaps it might be > possible in a future Aggregate release to automatically delete records > with duplicate IDs? > > I look forward to the Aggregate version with the SQL argument in the > body. > > The help is appreciated. > > Best, > David > > > On Apr 2, 5:07 am, Mitch Sundt wrote: >> Short answer: I Don't know. >> You should verify that you can upload submissions and view them OK on >> Aggregate. >> >> Location should have a type of 'geopoint' e.g., >> >> >> >> >> >> >> >> On Fri, Apr 1, 2011 at 1:55 PM, David wrote: >> > Thanks again for the help. How should the location field be corrected >> > in the XML? >> >> > Best, >> > David >> >> > On Apr 2, 3:56 am, Mitch Sundt wrote: >> > > Thanks for the additional sleuthing! >> >> > > I am not certain why yours did not work and eIMCI did, as they are both >> > > large surveys. >> >> > > I have confirmed through testing that the initial issue is that the URL >> > is >> > > too long >> > > and that by moving the 'sql' argument into the body, it successfully >> > > creates the table. There is a secondary issue with the >> > > location-fields/location >> > > field in your form which appears to not go through properly when >> > submitting >> > > data. I suspect this is because you don't gather that field data as a >> > > geopoint, yet >> > > you have the type defined as a geopoint. >> >> > > The final fix requires that I write a library. Working on that now. >> >> > > The submission of duplicates is a long-standing issue. The OpenRosa >> > > community >> > > is defining a 'metadata' section for compliant forms that contains an >> > > 'instanceId' field >> > > that would be a unique id that servers could use to identify repeat >> > > submissions. The >> > > support for this is not yet available. >> >https://bitbucket.org/javarosa/javarosa/wiki/OpenRosaMetaDataSchema >> >> > > Without that, there is only the manual comparisons of data rows and the >> > > manual deletions of >> > > the duplicates. Recording start and end times and the imei of the >> > > submitting phone >> > > can be used to simplify the identification of duplicate entries. >> >> > > Mitch >> >> > > On Thu, Mar 31, 2011 at 6:20 PM, David wrote: >> > > > Mitch, >> >> > > > Here is one more observation. On your demo server (http:// >> > > > opendatakit.appspot.com/forms?null), the form named "eIMCI by D-Tree" >> > > > has an even longer number of characters in the field titles (2575 to >> > > > be exact) than does my form (which has 2353). However, it can >> > > > successfully establish a connection to Fusion Tables. So, does that >> > > > mean that there could be another problem at hand? Thanks again. >> >> > > > Best, >> > > > David >> >> > > > On Apr 1, 7:41 am, David wrote: >> > > > > Mitch, >> >> > > > > Thanks for the response! In case it makes a difference, I note that >> > > > > even when I shorten the field names, the same error is still >> > > > > returned. I tried shortening the names quite a bit, and this >> > resulted >> > > > > in an error with a slightly shorter reported URL. >> >> > > > > How long might this take to fix in Aggregate? >> >> > > > > Also, I've had a problem on slow internet connections in which ODK >> > > > > Collect reports that there is failure in submitting records to >> > > > > Aggregate (0.95 on appspot), but it actually turns out that they have >> > > > > been at least partially transmitted. When one resubmits, this then >> > > > > results in duplicates of the same record in Aggregate. Is there a >> > fix >> > > > > for this? >> >> > > > > Best, >> > > > > David >> >> > > > > On Apr 1, 2:00 am, Mitch Sundt wrote: >> >> > > > > > Hi David, >> >> > > > > > We are passing the sql=CREATE... parameter as an argument on the >> > POST >> > > > > > request's URL, instead of in the body of the request, and that is >> > > > making the >> > > > > > URL string too long. It looks like FusionTables supports passing >> > the >> > > > > > parameter in the body of the POST request, which should eliminate >> > this >> > > > > > limitation. >> >> > > > > > I'll see what we can do to change this. >> >> > > > > > Mitch >> >> > > > > > On Thu, Mar 31, 2011 at 7:00 AM, David A. Raitzer wrote: >> >> > > > > > > Dear all, >> >> > > > > > > Any help that can be offered would be much appreciated. When I >> > try >> > > > to set up an ODK Aggregate (v.0.95) connection to Google Fusion tables, >> > the >> > > > error below is returned. I've tried shortening field names, changing >> > survey >> > > > IDs, and nothing seems to help. ODK can set up a connection to Google >> > > > Spreadsheets, but no data are transmitted other than submission times. >> >> > > > > > > What is the problem? The XML is attached for reference. >> >> > > > > > > Thanks in advance. >> >> > > > > > > Best, >> > > > > > > David >> >> > > > > > > Uncaught exception from servlet >> > > > > > > java.net.MalformedURLException: Invalid URL: >> > > >http://tables.googlelabs.com/api/query?sql=CREATE+TABLE+%27AdoptionSu. >> > .. >> >> > > > > > > at >> >> > com.google.appengine.api.urlfetch.URLFetchServiceImpl.convertApplicationExc >> > eption(URLFetchServiceImpl.java:106) >> > > > > > > at >> >> > com.google.appengine.api.urlfetch.URLFetchServiceImpl.fetch(URLFetchService >> > Impl.java:41) >> > > > > > > at >> >> > com.google.apphosting.utils.security.urlfetch.URLFetchServiceStreamHandler$ >> > Connection.fetchResponse(URLFetchServiceStreamHandler.java:418) >> >> > > > > > > at >> >> > com.google.apphosting.utils.security.urlfetch.URLFetchServiceStreamHandler$ >> > Connection.getInputStream(URLFetchServiceStreamHandler.java:297) >> > > > > > > at >> >> > org.odk.aggregate.form.remoteserver.FusionTableOAuth.executeInsert(FusionTa >> > bleOAuth.java:149) >> >> > > > > > > at >> >> > org.odk.aggregate.servlet.FusionTableServlet.doGet(FusionTableServlet.java: >> > 140) >> > > > > > > at >> > javax.servlet.http.HttpServlet.service(HttpServlet.java:617) >> > > > > > > at >> > javax.servlet.http.HttpServlet.service(HttpServlet.java:717) >> >> > > > > > > at >> > > > org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:511) >> > > > > > > at >> >> > org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandle >> > r.java:1166) >> > > > > > > at >> >> > com.google.apphosting.utils.servlet.ParseBlobUploadFilter.doFilter(ParseBlo >> > bUploadFilter.java:97) >> >> > > > > > > at >> >> > org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandle >> > r.java:1157) >> > > > > > > at >> >> > com.google.apphosting.runtime.jetty.SaveSessionFilter.doFilter(SaveSessionF >> > ilter.java:35) >> > > > > > > at >> >> > org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandle >> > r.java:1157) >> >> > > > > > > at >> >> > com.google.apphosting.utils.servlet.TransactionCleanupFilter.doFilter(Trans >> > actionCleanupFilter.java:43) >> > > > > > > at >> >> > org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandle >> > r.java:1157) >> > > > > > > at >> >> > org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:388) >> >> > > > > > > at >> >> > org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216) >> > > > > > > at >> >> > org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:182) >> > > > > > > at >> >> > org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:765) >> >> > > > > > > at >> > > > org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:418) >> > > > > > > at >> >> > com.google.apphosting.runtime.jetty.AppVersionHandlerMap.handle(AppVersionH >> > andlerMap.java:238) >> > > > > > > at >> >> > org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152) >> >> > > > > > > at org.mortbay.jetty.Server.handle(Server.java:326) >> > > > > > > at >> > > > org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:542) >> > > > > > > at >> >> > org.mortbay.jetty.HttpConnection$RequestHandler.headerComplete(HttpConnecti >> > on.java:923) >> >> > > > > > > at >> >> > com.google.apphosting.runtime.jetty.RpcRequestParser.parseAvailable(RpcRequ >> > estParser.java:76) >> > > > > > > at >> > > > org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:404) >> > > > > > > at >> >> > com.google.apphosting.runtime.jetty.JettyServletEngineAdapter.serviceReques >> > t(JettyServletEngineAdapter.java:135) >> >> > > > > > > at >> >> > com.google.apphosting.runtime.JavaRuntime.handleRequest(JavaRuntime.java:26 >> > 1) >> > > > > > > at >> >> > com.google.apphosting.base.RuntimePb$EvaluationRuntime$2.handleRequest(Runt >> > imePb.java:9285) >> > > > > > > at >> > > > com.google.net.rpc.impl.RpcUtil.runRpcInApplication(RpcUtil.java:437) >> >> > > > > > > at >> > > > com.google.net.rpc.impl.Server$RpcTask.runInContext(Server.java:573) >> > > > > > > at >> >> > com.google.tracing.TraceContext$TraceContextRunnable$1.run(TraceContext.jav >> > a:448) >> > > > > > > at >> > > > com.google.tracing.TraceContext.runInContext(TraceContext.java:688) >> >> > > > > > > at >> >> > com.google.tracing.TraceContext$AbstractTraceContextCallback.runInInherited >> > ContextNoUnref(TraceContext.java:326) >> > > > > > > at >> >> > com.google.tracing.TraceContext$AbstractTraceContextCallback.runInInherited >> > Context(TraceContext.java:318) >> >> > > > > > > Unexpected exception from servlet: >> > java.net.MalformedURLException: >> > > > Invalid URL: >> >> > > > > > ... >> >> > > > > > read more » >> >> > > > -- >> > > > Post: opendatakit@googlegroups.com >> > > > Unsubscribe: opendatakit+unsubscribe@googlegroups.com >> > > > Options:http://groups.google.com/group/opendatakit?hl=en >> >> > > -- >> > > Mitch Sundt >> > > Software Engineerhttp://www.OpenDataKit.org >> > > University of Washington >> > > mitchellsu...@gmail.com >> >> > -- >> > Post: opendatakit@googlegroups.com >> > Unsubscribe: opendatakit+unsubscribe@googlegroups.com >> > Options:http://groups.google.com/group/opendatakit?hl=en >> >> -- >> Mitch Sundt >> Software Engineerhttp://www.OpenDataKit.org >> University of Washington >> mitchellsu...@gmail.com- Hide quoted text - >> >> - Show quoted text - > > -- > Post: opendatakit@googlegroups.com > Unsubscribe: opendatakit+unsubscribe@googlegroups.com > Options: http://groups.google.com/group/opendatakit?hl=en >

David · April 2, 2011, 12:06pm

Waylon,

Thanks for your help. I am running this in the cloud, but am more
comfortable with an automatic configuration, if possible. Could it be
possible to email a copy of compiled version that can be automatically
uploaded to Appspot, if this will take longer than Tuesday?

Incidentally, is there a URL that can be used to pull a KML file
directly from Aggregate, given that this is managed by a GET request?
I tried using the following, with little success:
http://xxxx.appspot.com/kml?geopointField=geographic-fields-location&titleField=adopted-technologies&imageField&numberSubmissions=200&odkID=Adoption

Many thanks!

Best,
David

···

On Apr 2, 3:37 pm, "W. Brunette" wrote: > David, > > I believe we have fixes for the problems you listed with the 0.9.x > code base. The solution is currently available in the Aggregate code > repository at the tip of the default branch. Unfortunately the library > needed to allow a post to FusionTable instead of get request works > fine when running in the cloud but does not work when running locally. > > We need to figure out how we are going to resolve this before pushing > a release. If you are in a hurry and want to run it on Google App > Engine in the cloud and are somewhat technically savy you can follow > Aggregate's advanced instructions and setup an eclipse project and > push the code yourself. If you aren't in a hurry and can wait until > early next week we should have a solution published. > > Regards, > Waylon > > > > On Fri, Apr 1, 2011 at 7:49 PM, David wrote: > > Mitch, > > > Thanks. The location type is geopoint, as indicated below, and I can > > see the uploaded GPS submissions in Aggregate. Is there some other > > problem then? > > > It might also be noted that a Google spreadsheet can be established, > > but most of the data do not actually transfer. Is this also due to > > the location issue? > > > On a side note, the following code works for generating a unique user > > id in Collect: > jr:preload="uid" jr:preloadParams="general" /> Perhaps it might be > > possible in a future Aggregate release to automatically delete records > > with duplicate IDs? > > > I look forward to the Aggregate version with the SQL argument in the > > body. > > > The help is appreciated. > > > Best, > > David > > > On Apr 2, 5:07 am, Mitch Sundt wrote: > >> Short answer: I Don't know. > >> You should verify that you can upload submissions and view them OK on > >> Aggregate. > > >> Location should have a type of 'geopoint' e.g., > > >> > > >> On Fri, Apr 1, 2011 at 1:55 PM, David wrote: > >> > Thanks again for the help. How should the location field be corrected > >> > in the XML? > > >> > Best, > >> > David > > >> > On Apr 2, 3:56 am, Mitch Sundt wrote: > >> > > Thanks for the additional sleuthing! > > >> > > I am not certain why yours did not work and eIMCI did, as they are both > >> > > large surveys. > > >> > > I have confirmed through testing that the initial issue is that the URL > >> > is > >> > > too long > >> > > and that by moving the 'sql' argument into the body, it successfully > >> > > creates the table. There is a secondary issue with the > >> > > location-fields/location > >> > > field in your form which appears to not go through properly when > >> > submitting > >> > > data. I suspect this is because you don't gather that field data as a > >> > > geopoint, yet > >> > > you have the type defined as a geopoint. > > >> > > The final fix requires that I write a library. Working on that now. > > >> > > The submission of duplicates is a long-standing issue. The OpenRosa > >> > > community > >> > > is defining a 'metadata' section for compliant forms that contains an > >> > > 'instanceId' field > >> > > that would be a unique id that servers could use to identify repeat > >> > > submissions. The > >> > > support for this is not yet available. > >> >https://bitbucket.org/javarosa/javarosa/wiki/OpenRosaMetaDataSchema > > >> > > Without that, there is only the manual comparisons of data rows and the > >> > > manual deletions of > >> > > the duplicates. Recording start and end times and the imei of the > >> > > submitting phone > >> > > can be used to simplify the identification of duplicate entries. > > >> > > Mitch > > >> > > On Thu, Mar 31, 2011 at 6:20 PM, David wrote: > >> > > > Mitch, > > >> > > > Here is one more observation. On your demo server (http:// > >> > > > opendatakit.appspot.com/forms?null), the form named "eIMCI by D-Tree" > >> > > > has an even longer number of characters in the field titles (2575 to > >> > > > be exact) than does my form (which has 2353). However, it can > >> > > > successfully establish a connection to Fusion Tables. So, does that > >> > > > mean that there could be another problem at hand? Thanks again. > > >> > > > Best, > >> > > > David > > >> > > > On Apr 1, 7:41 am, David wrote: > >> > > > > Mitch, > > >> > > > > Thanks for the response! In case it makes a difference, I note that > >> > > > > even when I shorten the field names, the same error is still > >> > > > > returned. I tried shortening the names quite a bit, and this > >> > resulted > >> > > > > in an error with a slightly shorter reported URL. > > >> > > > > How long might this take to fix in Aggregate? > > >> > > > > Also, I've had a problem on slow internet connections in which ODK > >> > > > > Collect reports that there is failure in submitting records to > >> > > > > Aggregate (0.95 on appspot), but it actually turns out that they have > >> > > > > been at least partially transmitted. When one resubmits, this then > >> > > > > results in duplicates of the same record in Aggregate. Is there a > >> > fix > >> > > > > for this? > > >> > > > > Best, > >> > > > > David > > >> > > > > On Apr 1, 2:00 am, Mitch Sundt wrote: > > >> > > > > > Hi David, > > >> > > > > > We are passing the sql=CREATE... parameter as an argument on the > >> > POST > >> > > > > > request's URL, instead of in the body of the request, and that is > >> > > > making the > >> > > > > > URL string too long. It looks like FusionTables supports passing > >> > the > >> > > > > > parameter in the body of the POST request, which should eliminate > >> > this > >> > > > > > limitation. > > >> > > > > > I'll see what we can do to change this. > > >> > > > > > Mitch > > >> > > > > > On Thu, Mar 31, 2011 at 7:00 AM, David A. Raitzer wrote: > > >> > > > > > > Dear all, > > >> > > > > > > Any help that can be offered would be much appreciated. When I > >> > try > >> > > > to set up an ODK Aggregate (v.0.95) connection to Google Fusion tables, > >> > the > >> > > > error below is returned. I've tried shortening field names, changing > >> > survey > >> > > > IDs, and nothing seems to help. ODK can set up a connection to Google > >> > > > Spreadsheets, but no data are transmitted other than submission times. > > >> > > > > > > What is the problem? The XML is attached for reference. > > >> > > > > > > Thanks in advance. > > >> > > > > > > Best, > >> > > > > > > David > > >> > > > > > > Uncaught exception from servlet > >> > > > > > > java.net.MalformedURLException: Invalid URL: > >> > > >http://tables.googlelabs.com/api/query?sql=CREATE+TABLE+%27AdoptionSu. > >> > .. > > >> > > > > > > at > > >> > com.google.appengine.api.urlfetch.URLFetchServiceImpl.convertApplicationExc > >> > eption(URLFetchServiceImpl.java:106) > >> > > > > > > at > > >> > com.google.appengine.api.urlfetch.URLFetchServiceImpl.fetch(URLFetchService > >> > Impl.java:41) > >> > > > > > > at > > >> > com.google.apphosting.utils.security.urlfetch.URLFetchServiceStreamHandler$ > >> > Connection.fetchResponse(URLFetchServiceStreamHandler.java:418) > > >> > > > > > > at > > >> > com.google.apphosting.utils.security.urlfetch.URLFetchServiceStreamHandler$ > >> > Connection.getInputStream(URLFetchServiceStreamHandler.java:297) > >> > > > > > > at > > >> > org.odk.aggregate.form.remoteserver.FusionTableOAuth.executeInsert(FusionTa > >> > bleOAuth.java:149) > > >> > > > > > > at > > >> > org.odk.aggregate.servlet.FusionTableServlet.doGet(FusionTableServlet.java: > >> > 140) > >> > > > > > > at > >> > javax.servlet.http.HttpServlet.service(HttpServlet.java:617) > >> > > > > > > at > >> > javax.servlet.http.HttpServlet.service(HttpServlet.java:717) > > >> > > > > > > at > >> > > > org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:511) > >> > > > > > > at > > >> > org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandle > >> > r.java:1166) > >> > > > > > > at > > >> > com.google.apphosting.utils.servlet.ParseBlobUploadFilter.doFilter(ParseBlo > >> > bUploadFilter.java:97) > > >> > > > > > > at > > >> > org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandle > >> > r.java:1157) > >> > > > > > > at > > >> > com.google.apphosting.runtime.jetty.SaveSessionFilter.doFilter(SaveSessionF > >> > ilter.java:35) > >> > > > > > > at > > >> > org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandle > >> > r.java:1157) > > >> > > > > > > at > > >> > com.google.apphosting.utils.servlet.TransactionCleanupFilter.doFilter(Trans > >> > actionCleanupFilter.java:43) > >> > > > > > > at > > >> > org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandle > >> > r.java:1157) > >> > > > > > > at > > >> > org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:388) > > >> > > > > > > at > > >> > org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216) > >> > > > > > > at > > >> > org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:182) > >> > > > > > > at > > >> > org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:765) > > >> > > > > > > at > >> > > > org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:418) > >> > > > > > > at > > >> > com.google.apphosting.runtime.jetty.AppVersionHandlerMap.handle(AppVersionH > >> > andlerMap.java:238) > >> > > > > > > at > > >> > org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152) > > >> > > > > > > at org.mortbay.jetty.Server.handle(Server.java:326) > >> > > > > > > at > >> > > > org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:542) > >> > > > > > > at > > >> > org.mortbay.jetty.HttpConnection$RequestHandler.headerComplete(HttpConnecti > >> > on.java:923) > > >> > > > > > > at > > >> > com.google.apphosting.runtime.jetty.RpcRequestParser.parseAvailable(RpcRequ > >> > estParser.java:76) > >> > > > > > > at > > ... > > read more »- Hide quoted text - > > - Show quoted text -

Mitch_Sundt · April 4, 2011, 5:07pm

The localhost failure is a bug in AppEngine SDK 1.4.3; it works in AppEngine
SDK 1.4.2.

http://code.google.com/p/googleappengine/issues/detail?id=4823

Affects Mac, Linux and PCs.

Mitch

···

On Sat, Apr 2, 2011 at 5:06 AM, David wrote:

Waylon,

Thanks for your help. I am running this in the cloud, but am more
comfortable with an automatic configuration, if possible. Could it be
possible to email a copy of compiled version that can be automatically
uploaded to Appspot, if this will take longer than Tuesday?

Incidentally, is there a URL that can be used to pull a KML file
directly from Aggregate, given that this is managed by a GET request?
I tried using the following, with little success:

http://xxxx.appspot.com/kml?geopointField=geographic-fields-location&titleField=adopted-technologies&imageField&numberSubmissions=200&odkID=Adoption

Many thanks!

Best,
David

On Apr 2, 3:37 pm, "W. Brunette" wbrune...@gmail.com wrote:

David,

I believe we have fixes for the problems you listed with the 0.9.x
code base. The solution is currently available in the Aggregate code
repository at the tip of the default branch. Unfortunately the library
needed to allow a post to FusionTable instead of get request works
fine when running in the cloud but does not work when running locally.

We need to figure out how we are going to resolve this before pushing
a release. If you are in a hurry and want to run it on Google App
Engine in the cloud and are somewhat technically savy you can follow
Aggregate's advanced instructions and setup an eclipse project and
push the code yourself. If you aren't in a hurry and can wait until
early next week we should have a solution published.

Regards,
Waylon

On Fri, Apr 1, 2011 at 7:49 PM, David david.rait...@gmail.com wrote:

Mitch,

Thanks. The location type is geopoint, as indicated below, and I can
see the uploaded GPS submissions in Aggregate. Is there some other
problem then?

It might also be noted that a Google spreadsheet can be established,
but most of the data do not actually transfer. Is this also due to
the location issue?

On a side note, the following code works for generating a unique user
id in Collect: Perhaps it might be
possible in a future Aggregate release to automatically delete records
with duplicate IDs?

I look forward to the Aggregate version with the SQL argument in the
body.

The help is appreciated.

Best,
David

On Apr 2, 5:07 am, Mitch Sundt msu...@cs.washington.edu wrote:

Short answer: I Don't know.
You should verify that you can upload submissions and view them OK on
Aggregate.

Location should have a type of 'geopoint' e.g.,

On Fri, Apr 1, 2011 at 1:55 PM, David david.rait...@gmail.com wrote:

Thanks again for the help. How should the location field be
corrected
in the XML?

Best,
David

On Apr 2, 3:56 am, Mitch Sundt msu...@cs.washington.edu wrote:

Thanks for the additional sleuthing!

I am not certain why yours did not work and eIMCI did, as they are
both
large surveys.

I have confirmed through testing that the initial issue is that
the URL
is
too long
and that by moving the 'sql' argument into the body, it
successfully
creates the table. There is a secondary issue with the
location-fields/location
field in your form which appears to not go through properly when
submitting
data. I suspect this is because you don't gather that field data
as a
geopoint, yet
you have the type defined as a geopoint.

The final fix requires that I write a library. Working on that
now.

The submission of duplicates is a long-standing issue. The
OpenRosa
community
is defining a 'metadata' section for compliant forms that contains
an
'instanceId' field
that would be a unique id that servers could use to identify
repeat
submissions. The
support for this is not yet available.
https://bitbucket.org/javarosa/javarosa/wiki/OpenRosaMetaDataSchema

Without that, there is only the manual comparisons of data rows
and the
manual deletions of
the duplicates. Recording start and end times and the imei of the
submitting phone
can be used to simplify the identification of duplicate entries.

Mitch

On Thu, Mar 31, 2011 at 6:20 PM, David david.rait...@gmail.com wrote:

Mitch,

Here is one more observation. On your demo server (http://
opendatakit.appspot.com/forms?null), the form named "eIMCI by
D-Tree"
has an even longer number of characters in the field titles
(2575 to
be exact) than does my form (which has 2353). However, it can
successfully establish a connection to Fusion Tables. So, does
that
mean that there could be another problem at hand? Thanks again.

Best,
David

On Apr 1, 7:41 am, David david.rait...@gmail.com wrote:

Mitch,

Thanks for the response! In case it makes a difference, I
note that
even when I shorten the field names, the same error is still
returned. I tried shortening the names quite a bit, and this
resulted
in an error with a slightly shorter reported URL.

How long might this take to fix in Aggregate?

Also, I've had a problem on slow internet connections in which
ODK
Collect reports that there is failure in submitting records to
Aggregate (0.95 on appspot), but it actually turns out that
they have
been at least partially transmitted. When one resubmits, this
then
results in duplicates of the same record in Aggregate. Is
there a
fix
for this?

Best,
David

On Apr 1, 2:00 am, Mitch Sundt msu...@cs.washington.edu wrote:

Hi David,

We are passing the sql=CREATE... parameter as an argument on
the
POST
request's URL, instead of in the body of the request, and
that is
making the
URL string too long. It looks like FusionTables supports
passing
the
parameter in the body of the POST request, which should
eliminate
this
limitation.

I'll see what we can do to change this.

Mitch

On Thu, Mar 31, 2011 at 7:00 AM, David A. Raitzer david.rait...@gmail.comwrote:

Dear all,

Any help that can be offered would be much appreciated.
When I
try
to set up an ODK Aggregate (v.0.95) connection to Google Fusion
tables,
the
error below is returned. I've tried shortening field names,
changing
survey
IDs, and nothing seems to help. ODK can set up a connection to
Google
Spreadsheets, but no data are transmitted other than submission
times.

What is the problem? The XML is attached for reference.

Thanks in advance.

Best,
David

Uncaught exception from servlet
java.net.MalformedURLException: Invalid URL:

http://tables.googlelabs.com/api/query?sql=CREATE+TABLE+'AdoptionSu.

..

at

com.google.appengine.api.urlfetch.URLFetchServiceImpl.convertApplicationExc

eption(URLFetchServiceImpl.java:106)

at

com.google.appengine.api.urlfetch.URLFetchServiceImpl.fetch(URLFetchService

Impl.java:41)

at

com.google.apphosting.utils.security.urlfetch.URLFetchServiceStreamHandler$

Connection.fetchResponse(URLFetchServiceStreamHandler.java:418)

at

com.google.apphosting.utils.security.urlfetch.URLFetchServiceStreamHandler$

Connection.getInputStream(URLFetchServiceStreamHandler.java:297)

at

org.odk.aggregate.form.remoteserver.FusionTableOAuth.executeInsert(FusionTa

bleOAuth.java:149)

at

org.odk.aggregate.servlet.FusionTableServlet.doGet(FusionTableServlet.java:

at
javax.servlet.http.HttpServlet.service(HttpServlet.java:617)
at
javax.servlet.http.HttpServlet.service(HttpServlet.java:717)

at

org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:511)

at

org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandle

r.java:1166)

at

com.google.apphosting.utils.servlet.ParseBlobUploadFilter.doFilter(ParseBlo

bUploadFilter.java:97)

at

org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandle

r.java:1157)

at

com.google.apphosting.runtime.jetty.SaveSessionFilter.doFilter(SaveSessionF

ilter.java:35)

at

org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandle

r.java:1157)

at

com.google.apphosting.utils.servlet.TransactionCleanupFilter.doFilter(Trans

actionCleanupFilter.java:43)

at

org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandle

r.java:1157)

at

org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:388)

at

org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216)

at

org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:182)

at

org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:765)

at

org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:418)

at

com.google.apphosting.runtime.jetty.AppVersionHandlerMap.handle(AppVersionH

andlerMap.java:238)

at

org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152)

at org.mortbay.jetty.Server.handle(Server.java:326)
at

org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:542)

at

org.mortbay.jetty.HttpConnection$RequestHandler.headerComplete(HttpConnecti

on.java:923)

at

com.google.apphosting.runtime.jetty.RpcRequestParser.parseAvailable(RpcRequ

estParser.java:76)

at

...

read more »- Hide quoted text -

Show quoted text -

--
Post: opendatakit@googlegroups.com
Unsubscribe: opendatakit+unsubscribe@googlegroups.com
Options: http://groups.google.com/group/opendatakit?hl=en

--
Mitch Sundt
Software Engineer
http://www.OpenDataKit.org
University of Washington
mitchellsundt@gmail.com

W_Brunette · April 4, 2011, 11:31pm

We have posted a new version of ODK Aggregate (v0.9.6) to address the
large forms sending to FusionTable issue. Please file an issue if you
continue to have problems.

Cheers,
Waylon

···

On Mon, Apr 4, 2011 at 10:07 AM, Mitch Sundt wrote: > The localhost failure is a bug in AppEngine SDK 1.4.3; it works in AppEngine > SDK 1.4.2. > > http://code.google.com/p/googleappengine/issues/detail?id=4823 > > Affects Mac, Linux and PCs. > > Mitch > > On Sat, Apr 2, 2011 at 5:06 AM, David wrote: >> >> Waylon, >> >> Thanks for your help. I am running this in the cloud, but am more >> comfortable with an automatic configuration, if possible. Could it be >> possible to email a copy of compiled version that can be automatically >> uploaded to Appspot, if this will take longer than Tuesday? >> >> Incidentally, is there a URL that can be used to pull a KML file >> directly from Aggregate, given that this is managed by a GET request? >> I tried using the following, with little success: >> >> http://xxxx.appspot.com/kml?geopointField=geographic-fields-location&titleField=adopted-technologies&imageField&numberSubmissions=200&odkID=Adoption >> >> Many thanks! >> >> Best, >> David >> >> >> On Apr 2, 3:37 pm, "W. Brunette" wrote: >> > David, >> > >> > I believe we have fixes for the problems you listed with the 0.9.x >> > code base. The solution is currently available in the Aggregate code >> > repository at the tip of the default branch. Unfortunately the library >> > needed to allow a post to FusionTable instead of get request works >> > fine when running in the cloud but does not work when running locally. >> > >> > We need to figure out how we are going to resolve this before pushing >> > a release. If you are in a hurry and want to run it on Google App >> > Engine in the cloud and are somewhat technically savy you can follow >> > Aggregate's advanced instructions and setup an eclipse project and >> > push the code yourself. If you aren't in a hurry and can wait until >> > early next week we should have a solution published. >> > >> > Regards, >> > Waylon >> > >> > >> > >> > On Fri, Apr 1, 2011 at 7:49 PM, David wrote: >> > > Mitch, >> > >> > > Thanks. The location type is geopoint, as indicated below, and I can >> > > see the uploaded GPS submissions in Aggregate. Is there some other >> > > problem then? >> > >> > > It might also be noted that a Google spreadsheet can be established, >> > > but most of the data do not actually transfer. Is this also due to >> > > the location issue? >> > >> > > On a side note, the following code works for generating a unique user >> > > id in Collect: > > > jr:preload="uid" jr:preloadParams="general" /> Perhaps it might be >> > > possible in a future Aggregate release to automatically delete records >> > > with duplicate IDs? >> > >> > > I look forward to the Aggregate version with the SQL argument in the >> > > body. >> > >> > > The help is appreciated. >> > >> > > Best, >> > > David >> > >> > > On Apr 2, 5:07 am, Mitch Sundt wrote: >> > >> Short answer: I Don't know. >> > >> You should verify that you can upload submissions and view them OK on >> > >> Aggregate. >> > >> > >> Location should have a type of 'geopoint' e.g., >> > >> > >> >> > >> > >> On Fri, Apr 1, 2011 at 1:55 PM, David wrote: >> > >> > Thanks again for the help. How should the location field be >> > >> > corrected >> > >> > in the XML? >> > >> > >> > Best, >> > >> > David >> > >> > >> > On Apr 2, 3:56 am, Mitch Sundt wrote: >> > >> > > Thanks for the additional sleuthing! >> > >> > >> > > I am not certain why yours did not work and eIMCI did, as they >> > >> > > are both >> > >> > > large surveys. >> > >> > >> > > I have confirmed through testing that the initial issue is that >> > >> > > the URL >> > >> > is >> > >> > > too long >> > >> > > and that by moving the 'sql' argument into the body, it >> > >> > > successfully >> > >> > > creates the table. There is a secondary issue with the >> > >> > > location-fields/location >> > >> > > field in your form which appears to not go through properly when >> > >> > submitting >> > >> > > data. I suspect this is because you don't gather that field data >> > >> > > as a >> > >> > > geopoint, yet >> > >> > > you have the type defined as a geopoint. >> > >> > >> > > The final fix requires that I write a library. Working on that >> > >> > > now. >> > >> > >> > > The submission of duplicates is a long-standing issue. The >> > >> > > OpenRosa >> > >> > > community >> > >> > > is defining a 'metadata' section for compliant forms that >> > >> > > contains an >> > >> > > 'instanceId' field >> > >> > > that would be a unique id that servers could use to identify >> > >> > > repeat >> > >> > > submissions. The >> > >> > > support for this is not yet available. >> > >> >https://bitbucket.org/javarosa/javarosa/wiki/OpenRosaMetaDataSchema >> > >> > >> > > Without that, there is only the manual comparisons of data rows >> > >> > > and the >> > >> > > manual deletions of >> > >> > > the duplicates. Recording start and end times and the imei of >> > >> > > the >> > >> > > submitting phone >> > >> > > can be used to simplify the identification of duplicate entries. >> > >> > >> > > Mitch >> > >> > >> > > On Thu, Mar 31, 2011 at 6:20 PM, David wrote: >> > >> > > > Mitch, >> > >> > >> > > > Here is one more observation. On your demo server (http:// >> > >> > > > opendatakit.appspot.com/forms?null), the form named "eIMCI by >> > >> > > > D-Tree" >> > >> > > > has an even longer number of characters in the field titles >> > >> > > > (2575 to >> > >> > > > be exact) than does my form (which has 2353). However, it can >> > >> > > > successfully establish a connection to Fusion Tables. So, does >> > >> > > > that >> > >> > > > mean that there could be another problem at hand? Thanks >> > >> > > > again. >> > >> > >> > > > Best, >> > >> > > > David >> > >> > >> > > > On Apr 1, 7:41 am, David wrote: >> > >> > > > > Mitch, >> > >> > >> > > > > Thanks for the response! In case it makes a difference, I >> > >> > > > > note that >> > >> > > > > even when I shorten the field names, the same error is still >> > >> > > > > returned. I tried shortening the names quite a bit, and this >> > >> > resulted >> > >> > > > > in an error with a slightly shorter reported URL. >> > >> > >> > > > > How long might this take to fix in Aggregate? >> > >> > >> > > > > Also, I've had a problem on slow internet connections in >> > >> > > > > which ODK >> > >> > > > > Collect reports that there is failure in submitting records >> > >> > > > > to >> > >> > > > > Aggregate (0.95 on appspot), but it actually turns out that >> > >> > > > > they have >> > >> > > > > been at least partially transmitted. When one resubmits, >> > >> > > > > this then >> > >> > > > > results in duplicates of the same record in Aggregate. Is >> > >> > > > > there a >> > >> > fix >> > >> > > > > for this? >> > >> > >> > > > > Best, >> > >> > > > > David >> > >> > >> > > > > On Apr 1, 2:00 am, Mitch Sundt wrote: >> > >> > >> > > > > > Hi David, >> > >> > >> > > > > > We are passing the sql=CREATE... parameter as an argument >> > >> > > > > > on the >> > >> > POST >> > >> > > > > > request's URL, instead of in the body of the request, and >> > >> > > > > > that is >> > >> > > > making the >> > >> > > > > > URL string too long. It looks like FusionTables supports >> > >> > > > > > passing >> > >> > the >> > >> > > > > > parameter in the body of the POST request, which should >> > >> > > > > > eliminate >> > >> > this >> > >> > > > > > limitation. >> > >> > >> > > > > > I'll see what we can do to change this. >> > >> > >> > > > > > Mitch >> > >> > >> > > > > > On Thu, Mar 31, 2011 at 7:00 AM, David A. Raitzer wrote: >> > >> > >> > > > > > > Dear all, >> > >> > >> > > > > > > Any help that can be offered would be much appreciated. >> > >> > > > > > > When I >> > >> > try >> > >> > > > to set up an ODK Aggregate (v.0.95) connection to Google Fusion >> > >> > > > tables, >> > >> > the >> > >> > > > error below is returned. I've tried shortening field names, >> > >> > > > changing >> > >> > survey >> > >> > > > IDs, and nothing seems to help. ODK can set up a connection >> > >> > > > to Google >> > >> > > > Spreadsheets, but no data are transmitted other than submission >> > >> > > > times. >> > >> > >> > > > > > > What is the problem? The XML is attached for reference. >> > >> > >> > > > > > > Thanks in advance. >> > >> > >> > > > > > > Best, >> > >> > > > > > > David >> > >> > >> > > > > > > Uncaught exception from servlet >> > >> > > > > > > java.net.MalformedURLException: Invalid URL: >> > >> > > >> > >> > > > >http://tables.googlelabs.com/api/query?sql=CREATE+TABLE+%27AdoptionSu. >> > >> > .. >> > >> > >> > > > > > > at >> > >> > >> > >> > >> > com.google.appengine.api.urlfetch.URLFetchServiceImpl.convertApplicationExc >> > >> > eption(URLFetchServiceImpl.java:106) >> > >> > > > > > > at >> > >> > >> > >> > >> > com.google.appengine.api.urlfetch.URLFetchServiceImpl.fetch(URLFetchService >> > >> > Impl.java:41) >> > >> > > > > > > at >> > >> > >> > >> > >> > com.google.apphosting.utils.security.urlfetch.URLFetchServiceStreamHandler$ >> > >> > Connection.fetchResponse(URLFetchServiceStreamHandler.java:418) >> > >> > >> > > > > > > at >> > >> > >> > >> > >> > com.google.apphosting.utils.security.urlfetch.URLFetchServiceStreamHandler$ >> > >> > Connection.getInputStream(URLFetchServiceStreamHandler.java:297) >> > >> > > > > > > at >> > >> > >> > >> > >> > org.odk.aggregate.form.remoteserver.FusionTableOAuth.executeInsert(FusionTa >> > >> > bleOAuth.java:149) >> > >> > >> > > > > > > at >> > >> > >> > >> > >> > org.odk.aggregate.servlet.FusionTableServlet.doGet(FusionTableServlet.java: >> > >> > 140) >> > >> > > > > > > at >> > >> > javax.servlet.http.HttpServlet.service(HttpServlet.java:617) >> > >> > > > > > > at >> > >> > javax.servlet.http.HttpServlet.service(HttpServlet.java:717) >> > >> > >> > > > > > > at >> > >> > > > >> > >> > > > org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:511) >> > >> > > > > > > at >> > >> > >> > >> > >> > org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandle >> > >> > r.java:1166) >> > >> > > > > > > at >> > >> > >> > >> > >> > com.google.apphosting.utils.servlet.ParseBlobUploadFilter.doFilter(ParseBlo >> > >> > bUploadFilter.java:97) >> > >> > >> > > > > > > at >> > >> > >> > >> > >> > org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandle >> > >> > r.java:1157) >> > >> > > > > > > at >> > >> > >> > >> > >> > com.google.apphosting.runtime.jetty.SaveSessionFilter.doFilter(SaveSessionF >> > >> > ilter.java:35) >> > >> > > > > > > at >> > >> > >> > >> > >> > org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandle >> > >> > r.java:1157) >> > >> > >> > > > > > > at >> > >> > >> > >> > >> > com.google.apphosting.utils.servlet.TransactionCleanupFilter.doFilter(Trans >> > >> > actionCleanupFilter.java:43) >> > >> > > > > > > at >> > >> > >> > >> > >> > org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandle >> > >> > r.java:1157) >> > >> > > > > > > at >> > >> > >> > >> > >> > org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:388) >> > >> > >> > > > > > > at >> > >> > >> > >> > >> > org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216) >> > >> > > > > > > at >> > >> > >> > >> > >> > org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:182) >> > >> > > > > > > at >> > >> > >> > >> > >> > org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:765) >> > >> > >> > > > > > > at >> > >> > > > >> > >> > > > org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:418) >> > >> > > > > > > at >> > >> > >> > >> > >> > com.google.apphosting.runtime.jetty.AppVersionHandlerMap.handle(AppVersionH >> > >> > andlerMap.java:238) >> > >> > > > > > > at >> > >> > >> > >> > >> > org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152) >> > >> > >> > > > > > > at org.mortbay.jetty.Server.handle(Server.java:326) >> > >> > > > > > > at >> > >> > > > >> > >> > > > org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:542) >> > >> > > > > > > at >> > >> > >> > >> > >> > org.mortbay.jetty.HttpConnection$RequestHandler.headerComplete(HttpConnecti >> > >> > on.java:923) >> > >> > >> > > > > > > at >> > >> > >> > >> > >> > com.google.apphosting.runtime.jetty.RpcRequestParser.parseAvailable(RpcRequ >> > >> > estParser.java:76) >> > >> > > > > > > at >> > >> > ... >> > >> > read more »- Hide quoted text - >> > >> > - Show quoted text - >> >> -- >> Post: opendatakit@googlegroups.com >> Unsubscribe: opendatakit+unsubscribe@googlegroups.com >> Options: http://groups.google.com/group/opendatakit?hl=en > > > > -- > Mitch Sundt > Software Engineer > http://www.OpenDataKit.org > University of Washington > mitchellsundt@gmail.com > > -- > Post: opendatakit@googlegroups.com > Unsubscribe: opendatakit+unsubscribe@googlegroups.com > Options: http://groups.google.com/group/opendatakit?hl=en >

David · April 6, 2011, 6:03pm

Excellent! I can confirm now that this works. Many thanks!

Best,
David

···

On Apr 5, 7:31 am, "W. Brunette" wrote: > We have posted a new version of ODK Aggregate (v0.9.6) to address the > large forms sending to FusionTable issue. Please file an issue if you > continue to have problems. > > Cheers, > Waylon > > On Mon, Apr 4, 2011 at 10:07 AM, Mitch Sundt wrote: > > The localhost failure is a bug in AppEngine SDK 1.4.3; it works in AppEngine > > SDK 1.4.2. > > >http://code.google.com/p/googleappengine/issues/detail?id=4823 > > > Affects Mac, Linux and PCs. > > > Mitch > > > On Sat, Apr 2, 2011 at 5:06 AM, David wrote: > > >> Waylon, > > >> Thanks for your help. I am running this in the cloud, but am more > >> comfortable with an automatic configuration, if possible. Could it be > >> possible to email a copy of compiled version that can be automatically > >> uploaded to Appspot, if this will take longer than Tuesday? > > >> Incidentally, is there a URL that can be used to pull a KML file > >> directly from Aggregate, given that this is managed by a GET request? > >> I tried using the following, with little success: > > >>http://xxxx.appspot.com/kml?geopointField=geographic-fields-location&... > > >> Many thanks! > > >> Best, > >> David > > >> On Apr 2, 3:37 pm, "W. Brunette" wrote: > >> > David, > > >> > I believe we have fixes for the problems you listed with the 0.9.x > >> > code base. The solution is currently available in the Aggregate code > >> > repository at the tip of the default branch. Unfortunately the library > >> > needed to allow a post to FusionTable instead of get request works > >> > fine when running in the cloud but does not work when running locally. > > >> > We need to figure out how we are going to resolve this before pushing > >> > a release. If you are in a hurry and want to run it on Google App > >> > Engine in the cloud and are somewhat technically savy you can follow > >> > Aggregate's advanced instructions and setup an eclipse project and > >> > push the code yourself. If you aren't in a hurry and can wait until > >> > early next week we should have a solution published. > > >> > Regards, > >> > Waylon > > >> > On Fri, Apr 1, 2011 at 7:49 PM, David wrote: > >> > > Mitch, > > >> > > Thanks. The location type is geopoint, as indicated below, and I can > >> > > see the uploaded GPS submissions in Aggregate. Is there some other > >> > > problem then? > > >> > > It might also be noted that a Google spreadsheet can be established, > >> > > but most of the data do not actually transfer. Is this also due to > >> > > the location issue? > > >> > > On a side note, the following code works for generating a unique user > >> > > id in Collect: >> > > jr:preload="uid" jr:preloadParams="general" /> Perhaps it might be > >> > > possible in a future Aggregate release to automatically delete records > >> > > with duplicate IDs? > > >> > > I look forward to the Aggregate version with the SQL argument in the > >> > > body. > > >> > > The help is appreciated. > > >> > > Best, > >> > > David > > >> > > On Apr 2, 5:07 am, Mitch Sundt wrote: > >> > >> Short answer: I Don't know. > >> > >> You should verify that you can upload submissions and view them OK on > >> > >> Aggregate. > > >> > >> Location should have a type of 'geopoint' e.g., > > >> > >> > > >> > >> On Fri, Apr 1, 2011 at 1:55 PM, David wrote: > >> > >> > Thanks again for the help. How should the location field be > >> > >> > corrected > >> > >> > in the XML? > > >> > >> > Best, > >> > >> > David > > >> > >> > On Apr 2, 3:56 am, Mitch Sundt wrote: > >> > >> > > Thanks for the additional sleuthing! > > >> > >> > > I am not certain why yours did not work and eIMCI did, as they > >> > >> > > are both > >> > >> > > large surveys. > > >> > >> > > I have confirmed through testing that the initial issue is that > >> > >> > > the URL > >> > >> > is > >> > >> > > too long > >> > >> > > and that by moving the 'sql' argument into the body, it > >> > >> > > successfully > >> > >> > > creates the table. There is a secondary issue with the > >> > >> > > location-fields/location > >> > >> > > field in your form which appears to not go through properly when > >> > >> > submitting > >> > >> > > data. I suspect this is because you don't gather that field data > >> > >> > > as a > >> > >> > > geopoint, yet > >> > >> > > you have the type defined as a geopoint. > > >> > >> > > The final fix requires that I write a library. Working on that > >> > >> > > now. > > >> > >> > > The submission of duplicates is a long-standing issue. The > >> > >> > > OpenRosa > >> > >> > > community > >> > >> > > is defining a 'metadata' section for compliant forms that > >> > >> > > contains an > >> > >> > > 'instanceId' field > >> > >> > > that would be a unique id that servers could use to identify > >> > >> > > repeat > >> > >> > > submissions. The > >> > >> > > support for this is not yet available. > >> > >> >https://bitbucket.org/javarosa/javarosa/wiki/OpenRosaMetaDataSchema > > >> > >> > > Without that, there is only the manual comparisons of data rows > >> > >> > > and the > >> > >> > > manual deletions of > >> > >> > > the duplicates. Recording start and end times and the imei of > >> > >> > > the > >> > >> > > submitting phone > >> > >> > > can be used to simplify the identification of duplicate entries. > > >> > >> > > Mitch > > >> > >> > > On Thu, Mar 31, 2011 at 6:20 PM, David wrote: > >> > >> > > > Mitch, > > >> > >> > > > Here is one more observation. On your demo server (http:// > >> > >> > > > opendatakit.appspot.com/forms?null), the form named "eIMCI by > >> > >> > > > D-Tree" > >> > >> > > > has an even longer number of characters in the field titles > >> > >> > > > (2575 to > >> > >> > > > be exact) than does my form (which has 2353). However, it can > >> > >> > > > successfully establish a connection to Fusion Tables. So, does > >> > >> > > > that > >> > >> > > > mean that there could be another problem at hand? Thanks > >> > >> > > > again. > > >> > >> > > > Best, > >> > >> > > > David > > >> > >> > > > On Apr 1, 7:41 am, David wrote: > >> > >> > > > > Mitch, > > >> > >> > > > > Thanks for the response! In case it makes a difference, I > >> > >> > > > > note that > >> > >> > > > > even when I shorten the field names, the same error is still > >> > >> > > > > returned. I tried shortening the names quite a bit, and this > >> > >> > resulted > >> > >> > > > > in an error with a slightly shorter reported URL. > > >> > >> > > > > How long might this take to fix in Aggregate? > > >> > >> > > > > Also, I've had a problem on slow internet connections in > >> > >> > > > > which ODK > >> > >> > > > > Collect reports that there is failure in submitting records > >> > >> > > > > to > >> > >> > > > > Aggregate (0.95 on appspot), but it actually turns out that > >> > >> > > > > they have > >> > >> > > > > been at least partially transmitted. When one resubmits, > >> > >> > > > > this then > >> > >> > > > > results in duplicates of the same record in Aggregate. Is > >> > >> > > > > there a > >> > >> > fix > >> > >> > > > > for this? > > >> > >> > > > > Best, > >> > >> > > > > David > > >> > >> > > > > On Apr 1, 2:00 am, Mitch Sundt wrote: > > >> > >> > > > > > Hi David, > > >> > >> > > > > > We are passing the sql=CREATE... parameter as an argument > >> > >> > > > > > on the > >> > >> > POST > >> > >> > > > > > request's URL, instead of in the body of the request, and > >> > >> > > > > > that is > >> > >> > > > making the > >> > >> > > > > > URL string too long. It looks like FusionTables supports > >> > >> > > > > > passing > >> > >> > the > >> > >> > > > > > parameter in the body of the POST request, which should > >> > >> > > > > > eliminate > >> > >> > this > >> > >> > > > > > limitation. > > >> > >> > > > > > I'll see what we can do to change this. > > >> > >> > > > > > Mitch > > >> > >> > > > > > On Thu, Mar 31, 2011 at 7:00 AM, David A. Raitzer wrote: > > >> > >> > > > > > > Dear all, > > >> > >> > > > > > > Any help that can be offered would be much appreciated. > >> > >> > > > > > > When I > >> > >> > try > >> > >> > > > to set up an ODK Aggregate (v.0.95) connection to Google Fusion > >> > >> > > > tables, > >> > >> > the > >> > >> > > > error below is returned. I've tried shortening field names, > >> > >> > > > changing > >> > >> > survey > >> > >> > > > IDs, and nothing seems to help. ODK can set up a connection > >> > >> > > > to Google > >> > >> > > > Spreadsheets, but no data are transmitted other than submission > >> > >> > > > times. > > >> > >> > > > > > > What is the problem? The XML is attached for reference. > > >> > >> > > > > > > Thanks in advance. > > >> > >> > > > > > > Best, > >> > >> > > > > > > David > > >> > >> > > > > > > Uncaught exception from servlet > >> > >> > > > > > > java.net.MalformedURLException: Invalid URL: > > >> > >> > > > >http://tables.googlelabs.com/api/query?sql=CREATE+TABLE+%27AdoptionSu. > >> > >> > .. > > >> > >> > > > > > > at > > >> > >> > com.google.appengine.api.urlfetch.URLFetchServiceImpl.convertApplicationExc > >> > >> > eption(URLFetchServiceImpl.java:106) > >> > >> > > > > > > at > > >> > >> > com.google.appengine.api.urlfetch.URLFetchServiceImpl.fetch(URLFetchService > >> > >> > Impl.java:41) > >> > >> > > > > > > at > > >> > >> > com.google.apphosting.utils.security.urlfetch.URLFetchServiceStreamHandler$ > >> > >> > Connection.fetchResponse(URLFetchServiceStreamHandler.java:418) > > >> > >> > > > > > > at > > >> > >> > com.google.apphosting.utils.security.urlfetch.URLFetchServiceStreamHandler$ > >> > >> > Connection.getInputStream(URLFetchServiceStreamHandler.java:297) > >> > >> > > > > > > at > > ... > > read more »

Mitch_Sundt · April 6, 2011, 6:53pm

Great!

Glad to hear that the problem is fixed.

Mitch

···

On Wed, Apr 6, 2011 at 11:03 AM, David wrote:

Excellent! I can confirm now that this works. Many thanks!

Best,
David

On Apr 5, 7:31 am, "W. Brunette" wbrune...@gmail.com wrote:

We have posted a new version of ODK Aggregate (v0.9.6) to address the
large forms sending to FusionTable issue. Please file an issue if you
continue to have problems.

Cheers,
Waylon

On Mon, Apr 4, 2011 at 10:07 AM, Mitch Sundt msu...@cs.washington.edu wrote:

The localhost failure is a bug in AppEngine SDK 1.4.3; it works in
AppEngine
SDK 1.4.2.

http://code.google.com/p/googleappengine/issues/detail?id=4823

Affects Mac, Linux and PCs.

Mitch

On Sat, Apr 2, 2011 at 5:06 AM, David david.rait...@gmail.com wrote:

Waylon,

Thanks for your help. I am running this in the cloud, but am more
comfortable with an automatic configuration, if possible. Could it be
possible to email a copy of compiled version that can be automatically
uploaded to Appspot, if this will take longer than Tuesday?

Incidentally, is there a URL that can be used to pull a KML file
directly from Aggregate, given that this is managed by a GET request?
I tried using the following, with little success:

http://xxxx.appspot.com/kml?geopointField=geographic-fields-location&.
..

Many thanks!

Best,
David

On Apr 2, 3:37 pm, "W. Brunette" wbrune...@gmail.com wrote:

David,

I believe we have fixes for the problems you listed with the 0.9.x
code base. The solution is currently available in the Aggregate code
repository at the tip of the default branch. Unfortunately the
library
needed to allow a post to FusionTable instead of get request works
fine when running in the cloud but does not work when running
locally.

We need to figure out how we are going to resolve this before
pushing
a release. If you are in a hurry and want to run it on Google App
Engine in the cloud and are somewhat technically savy you can follow
Aggregate's advanced instructions and setup an eclipse project and
push the code yourself. If you aren't in a hurry and can wait until
early next week we should have a solution published.

Regards,
Waylon

On Fri, Apr 1, 2011 at 7:49 PM, David david.rait...@gmail.com wrote:

Mitch,

Thanks. The location type is geopoint, as indicated below, and I
can
see the uploaded GPS submissions in Aggregate. Is there some
other
problem then?

It might also be noted that a Google spreadsheet can be
established,
but most of the data do not actually transfer. Is this also due
to
the location issue?

On a side note, the following code works for generating a unique
user
id in Collect: Perhaps it might
be
possible in a future Aggregate release to automatically delete
records
with duplicate IDs?

I look forward to the Aggregate version with the SQL argument in
the
body.

The help is appreciated.

Best,
David

On Apr 2, 5:07 am, Mitch Sundt msu...@cs.washington.edu wrote:

Short answer: I Don't know.
You should verify that you can upload submissions and view them
OK on
Aggregate.

Location should have a type of 'geopoint' e.g.,

On Fri, Apr 1, 2011 at 1:55 PM, David david.rait...@gmail.com wrote:

Thanks again for the help. How should the location field be
corrected
in the XML?

Best,
David

On Apr 2, 3:56 am, Mitch Sundt msu...@cs.washington.edu wrote:

Thanks for the additional sleuthing!

I am not certain why yours did not work and eIMCI did, as
they
are both
large surveys.

I have confirmed through testing that the initial issue is
that
the URL
is
too long
and that by moving the 'sql' argument into the body, it
successfully
creates the table. There is a secondary issue with the
location-fields/location
field in your form which appears to not go through properly
when
submitting
data. I suspect this is because you don't gather that field
data
as a
geopoint, yet
you have the type defined as a geopoint.

The final fix requires that I write a library. Working on
that
now.

The submission of duplicates is a long-standing issue. The
OpenRosa
community
is defining a 'metadata' section for compliant forms that
contains an
'instanceId' field
that would be a unique id that servers could use to identify
repeat
submissions. The
support for this is not yet available.

https://bitbucket.org/javarosa/javarosa/wiki/OpenRosaMetaDataSchema

Without that, there is only the manual comparisons of data
rows
and the
manual deletions of
the duplicates. Recording start and end times and the imei
of
the
submitting phone
can be used to simplify the identification of duplicate
entries.

Mitch

On Thu, Mar 31, 2011 at 6:20 PM, David < david.rait...@gmail.com> wrote:

Mitch,

Here is one more observation. On your demo server (http://
opendatakit.appspot.com/forms?null), the form named "eIMCI
by
D-Tree"
has an even longer number of characters in the field titles
(2575 to
be exact) than does my form (which has 2353). However, it
can
successfully establish a connection to Fusion Tables. So,
does
that
mean that there could be another problem at hand? Thanks
again.

Best,
David

On Apr 1, 7:41 am, David david.rait...@gmail.com wrote:

Mitch,

Thanks for the response! In case it makes a difference,
I
note that
even when I shorten the field names, the same error is
still
returned. I tried shortening the names quite a bit, and
this
resulted
in an error with a slightly shorter reported URL.

How long might this take to fix in Aggregate?

Also, I've had a problem on slow internet connections in
which ODK
Collect reports that there is failure in submitting
records
to
Aggregate (0.95 on appspot), but it actually turns out
that
they have
been at least partially transmitted. When one resubmits,
this then
results in duplicates of the same record in Aggregate.
Is
there a
fix
for this?

Best,
David

On Apr 1, 2:00 am, Mitch Sundt <msu...@cs.washington.edu wrote:

Hi David,

We are passing the sql=CREATE... parameter as an
argument
on the
POST
request's URL, instead of in the body of the request,
and
that is
making the
URL string too long. It looks like FusionTables
supports
passing
the
parameter in the body of the POST request, which should
eliminate
this
limitation.

I'll see what we can do to change this.

Mitch

On Thu, Mar 31, 2011 at 7:00 AM, David A. Raitzer david.rait...@gmail.comwrote:

Dear all,

Any help that can be offered would be much
appreciated.
When I
try
to set up an ODK Aggregate (v.0.95) connection to Google
Fusion
tables,
the
error below is returned. I've tried shortening field
names,
changing
survey
IDs, and nothing seems to help. ODK can set up a
connection
to Google
Spreadsheets, but no data are transmitted other than
submission
times.

What is the problem? The XML is attached for
reference.

Thanks in advance.

Best,
David

Uncaught exception from servlet
java.net.MalformedURLException: Invalid URL:

http://tables.googlelabs.com/api/query?sql=CREATE+TABLE+'AdoptionSu.

..

at

com.google.appengine.api.urlfetch.URLFetchServiceImpl.convertApplicationExc

eption(URLFetchServiceImpl.java:106)

at

com.google.appengine.api.urlfetch.URLFetchServiceImpl.fetch(URLFetchService

Impl.java:41)

at

com.google.apphosting.utils.security.urlfetch.URLFetchServiceStreamHandler$

Connection.fetchResponse(URLFetchServiceStreamHandler.java:418)

at

com.google.apphosting.utils.security.urlfetch.URLFetchServiceStreamHandler$

Connection.getInputStream(URLFetchServiceStreamHandler.java:297)

at

...

read more »

--
Post: opendatakit@googlegroups.com
Unsubscribe: opendatakit+unsubscribe@googlegroups.com
Options: http://groups.google.com/group/opendatakit?hl=en

--
Mitch Sundt
Software Engineer
http://www.OpenDataKit.org
University of Washington
mitchellsundt@gmail.com