ODK Aggregate 1.6.1 Submission error

Support
1

Hello ODK Developers,

I briefly tried out ODK Aggregate but something seem not to work fine. Downloads work ok but uploads seem to have a problem. I get this error from calatina out;

INFO: Inside doHead
Aug 24, 2018 5:32:31 AM org.opendatakit.aggregate.servlet.SubmissionServlet doPost
SEVERE: Unexpected exception: null
java.net.SocketTimeoutException
	at org.apache.tomcat.util.net.NioBlockingSelector.read(NioBlockingSelector.java:202)
	at org.apache.tomcat.util.net.NioSelectorPool.read(NioSelectorPool.java:250)
	at org.apache.tomcat.util.net.NioSelectorPool.read(NioSelectorPool.java:231)
	at org.apache.coyote.http11.InternalNioInputBuffer.fill(InternalNioInputBuffer.java:133)
	at org.apache.coyote.http11.InternalNioInputBuffer$SocketInputBuffer.doRead(InternalNioInputBuffer.java:177)
	at org.apache.coyote.http11.filters.IdentityInputFilter.doRead(IdentityInputFilter.java:110)
	at org.apache.coyote.http11.AbstractInputBuffer.doRead(AbstractInputBuffer.java:362)
	at org.apache.coyote.Request.doRead(Request.java:476)
	at org.apache.catalina.connector.InputBuffer.realReadBytes(InputBuffer.java:350)
	at org.apache.tomcat.util.buf.ByteChunk.substract(ByteChunk.java:395)
	at org.apache.catalina.connector.InputBuffer.read(InputBuffer.java:375)
	at org.apache.catalina.connector.CoyoteInputStream.read(CoyoteInputStream.java:190)
	at org.apache.commons.fileupload.MultipartStream$ItemInputStream.makeAvailable(MultipartStream.java:999)
	at org.apache.commons.fileupload.MultipartStream$ItemInputStream.read(MultipartStream.java:903)
	at java.io.FilterInputStream.read(FilterInputStream.java:133)
	at org.apache.commons.fileupload.util.LimitedInputStream.read(LimitedInputStream.java:134)
	at java.io.BufferedInputStream.fill(BufferedInputStream.java:246)
	at java.io.BufferedInputStream.read(BufferedInputStream.java:265)
	at org.opendatakit.aggregate.parser.MultiPartFormData.<init>(MultiPartFormData.java:100)
	at org.opendatakit.aggregate.servlet.SubmissionServlet.doPost(SubmissionServlet.java:224)
	at javax.servlet.http.HttpServlet.service(HttpServlet.java:648)
	at javax.servlet.http.HttpServlet.service(HttpServlet.java:729)
	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:292)
	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:207)
	at org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:52)
	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:240)
	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:207)
	at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:317)
	at org.springframework.security.web.access.intercept.FilterSecurityInterceptor.invoke(FilterSecurityInterceptor.java:127)
	at org.springframework.security.web.access.intercept.FilterSecurityInterceptor.doFilter(FilterSecurityInterceptor.java:91)
	at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331)
	at org.springframework.security.web.access.ExceptionTranslationFilter.doFilter(ExceptionTranslationFilter.java:114)
	at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331)
	at org.springframework.security.web.session.SessionManagementFilter.doFilter(SessionManagementFilter.java:137)
	at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331)
	at org.opendatakit.common.security.spring.SecurityContextHolderAwareAuthPreservingRequestFilter.doFilter(SecurityContextHolderAwareAuthPreservingRequestFilter.java:66)
	at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331)
	at org.springframework.security.web.authentication.AnonymousAuthenticationFilter.doFilter(AnonymousAuthenticationFilter.java:111)
	at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331)
	at org.springframework.security.web.savedrequest.RequestCacheAwareFilter.doFilter(RequestCacheAwareFilter.java:63)
	at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331)
	at org.opendatakit.common.security.spring.DigestAuthenticationFilter.doFilter(DigestAuthenticationFilter.java:42)
	at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331)
	at org.opendatakit.common.security.spring.OutOfBandUserFilter.doFilter(OutOfBandUserFilter.java:105)
	at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331)
	at org.opendatakit.common.security.spring.Oauth2ResourceFilter.doFilter(Oauth2ResourceFilter.java:352)
	at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331)
	at org.springframework.security.web.authentication.logout.LogoutFilter.doFilter(LogoutFilter.java:116)
	at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331)
	at org.springframework.security.web.context.SecurityContextPersistenceFilter.doFilter(SecurityContextPersistenceFilter.java:105)
	at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331)
	at org.springframework.security.web.access.channel.ChannelProcessingFilter.doFilter(ChannelProcessingFilter.java:157)
	at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331)
	at org.springframework.security.web.FilterChainProxy.doFilterInternal(FilterChainProxy.java:214)
	at org.springframework.security.web.FilterChainProxy.doFilter(FilterChainProxy.java:177)
	at org.springframework.web.filter.DelegatingFilterProxy.invokeDelegate(DelegatingFilterProxy.java:346)
	at org.springframework.web.filter.DelegatingFilterProxy.doFilter(DelegatingFilterProxy.java:262)
	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:240)
	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:207)
	at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:212)
	at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:94)
	at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:504)
	at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:141)
	at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:79)
	at org.apache.catalina.valves.AbstractAccessLogValve.invoke(AbstractAccessLogValve.java:620)
	at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:88)
	at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:509)
	at org.apache.coyote.http11.AbstractHttp11Processor.process(AbstractHttp11Processor.java:1104)
	at org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:684)
	at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1524)
	at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.run(NioEndpoint.java:1480)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
	at java.lang.Thread.run(Thread.java:748)

This happens when one successful submission has happened and you are attempting to upload the next submit a second form. The successful submissions are fast.

Paul

2

Your post does not contain enough information for me to answer your question. In general, the best way to get help is to fill out the questions in the pre-filled template that you get when you create a support post. So, with that in mind, can you answer these questions?

  1. What ODK version are you using? And on what operating system version?
  2. What steps can we take to reproduce the problem?
  3. What you have you tried to fix the problem?
3

@yanokwa,

  1. I am running ODK Aggregate 1.6.1. The Android devices are running Collect 1.16.3
  2. The Server is running on CentOs 7
  3. To fix the problem, I installed Aggregate 1.5 and it works perfect. I checked on network speed and all is fine. I also tried to upload data on the LAN/WiFi but was unsuccessful. I suspected a larger form was the problem and designed a small form but submissions failed too.
  4. On the Android device side, error 500 is displayed.

Paul

4

Thanks, @paul_macharia! I'll review your stacktrace tomorrow. It would help me to have the form and a pair of submissions (ideally the same you have tried) to do my research and try to reproduce your same issue here. Could you send them to ggalmazor@gmail.com, please?

5

Hi, @paul_macharia!

I have tried two things:

  • I've deployed Aggregate v1.6.1 locally in a Tomcat8 server and uploaded the two forms you have sent me by email.

    • I've used Collect to create a pair of submissions and send them.
    • I haven't got any errors. Both submissions in both forms were saved correctly on Aggregate

  • I've analyzed the stacktrace you have attached in your original post. Some insights from that:

    • The part of Aggregate that has reported the error hasn't been changed since before v1.5.0. This means that when you got to send the submissions with v1.5.0, the same code was running when that happened.
    • The actual error says there's been a socket timeout, meaning that communication between the mobile phone and the server has been interrupted for some reason.

All these findings are coherent with a networking problem at the moment of submitting the submissions.
Did you try get the same error consistently when trying to send the second submission?

Also, I've uploaded the forms to https://aggregate-test-2.appspot.com. Could you try to send some submissions with fake data to see if we get to reproduce the error there?

6

Thanks @ggalmazor,

I very much appreciate this feedback. I will test this on your appspot. I could also redeploy the instance on our servers and allow you to attempt upload data so that we see how it works out.

Paul

1 Like
7

I could also redeploy the instance on our servers and allow you to attempt upload data so that we see how it works out.

That could be useful too :slight_smile:

1 Like
8

Hi, @paul_macharia!

I just want to document the new findings we've talked about by email.

In the screenshot you sent me, we could see that connection attempts to the server where being rejected. It seems like your issue is related to some networking problem, maybe to a misconfiguration of a firewall software that could be blacklisting requests.

Unfortunately, I don't think I can be of much help in this particular scenario. Maybe there's some other user that has faced networking/firewall issues and can throw some light on this.

9

Thanks @ggalmazor,

I have been working on this with network experts and this is their observation (This is after I did a complete system setup. Installed new OS, Tomcat 8, MariaDB 10 and Aggregate 1.6.1)

"From the firewall log analysis, am now seeing some TCP Resets initiated from the clients (PFA). I would advise that you also check the client application stack, there is something amiss between the client and server applications, probably around session timeouts"

Would this provide more details of the problem?

Paul

10

@ggalmazor,

Its not 1.5 or 1.6 specific. I downgraded from 1.6 to 1.5 but the problem started happening on the system.

1 Like