Reoccurring Aggregate UploadSubmissionsWorkerImpl Error Causing High GAE Charges

Support
1

What is the problem? Please be detailed.

Per the screenshot below, we are having a reoccurring error with a frequency of roughly 2.6 minutes (n = 547 in last 24 hours) on the background service. I expanded the period of the logs to 30 days, and it has been happening for at least the last month with roughly 500 to 600 errors per day. We had heavy traffic on our server in late September and early October, but things are pretty calm right now.

I believe this may also be linked to a recent increase in our GAE charges, which have been 3x to 5x higher in the last month than in previous months. Since Nov 1, the substantial charges are for App Engine Cloud Datastore Read Ops (95236131), App Engine Frontend Instances (378 hours), and App Engine Backend instances (338 hours).

@ggalmazor I don't know if this is linked to the other issue we were talking about or not, but it appears that this is an urgent issue.

image
image1787×786 102 KB

When looking at the App Engine summary, there is some persistent task taking roughly 0.2 counts/sec that isn't shown as either "Client" or "Server" but "Total." The request type is "Dynamic."

image
image824×340 21.7 KB

What ODK tool and version are you using? And on what device and operating system version?

Aggregate v1.4.15 of GAE

What you have you tried to fix the problem?

Per an earlier post, I've disabled faster background actions around 9 a.m. CET (8 a.m. UTM), and that seems to have stopped the error (and hopefully the charges) for now. This also brought the counts/sec for Total to zero.

image
image944×233 5.74 KB

Anything else we should know or have? If you have a test form or screenshots or logs, attach here.

org.opendatakit.aggregate.task.UploadSubmissionsWorkerImpl: uploadAllSubmissions: org.opendatakit.aggregate.exception.ODKExternalServiceException: java.lang.IllegalStateException: java.net.URISyntaxException: Illegal character in path at index 30: http://35.192.48.24:8089/entry
at org.opendatakit.aggregate.externalservice.JsonServer.sendRequest (JsonServer.java:148)
at org.opendatakit.aggregate.externalservice.JsonServer.insertData (JsonServer.java:197)
at org.opendatakit.aggregate.externalservice.AbstractExternalService.sendSubmission (AbstractExternalService.java:138)
at org.opendatakit.aggregate.task.UploadSubmissionsWorkerImpl.sendSubmissions (UploadSubmissionsWorkerImpl.java:322)
at org.opendatakit.aggregate.task.UploadSubmissionsWorkerImpl.uploadSubmissions (UploadSubmissionsWorkerImpl.java:282)
at org.opendatakit.aggregate.task.UploadSubmissionsWorkerImpl.uploadAllSubmissions (UploadSubmissionsWorkerImpl.java:196)
at org.opendatakit.aggregate.task.gae.servlet.UploadSubmissionsTaskServlet.doGet (UploadSubmissionsTaskServlet.java:109)
at org.opendatakit.common.security.spring.SecurityContextHolderAwareAuthPreservingRequestFilter.doFilter (SecurityContextHolderAwareAuthPreservingRequestFilter.java:66)
at org.opendatakit.common.security.spring.DigestAuthenticationFilter.doFilter (DigestAuthenticationFilter.java:40)
at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter (FilterChainProxy.java:331)
at org.opendatakit.common.security.spring.OutOfBandUserFilter.doFilter (OutOfBandUserFilter.java:105)
at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter (FilterChainProxy.java:331)
at org.opendatakit.common.security.spring.Oauth2ResourceFilter.doFilter (Oauth2ResourceFilter.java:352)
at io.grpc.Context.run (Context.java:565)

2

Hi @Jeff_Davids. I'm on it!

3

Hi @Jeff_Davids!

For what I can gather, at least one of your configured Json Server publisher is producing those errors because there's an extra trailing whitespace in the server url field of the publisher. Instead of "http://111.222.333.444:1234/entry" you've got "http://111.222.333.444:1234/entry ".

I have been able to reproduce your error on my local Aggregate by inserting such a trailing whitespace.

If you want, I could try to reset the publishers for you (I still have access to your Google Cloud project) ensuring that there's no trailing whitespace.

4

Hi @ggalmazor! That would be amazing if you could take a look and see if you can find the trailing whitespace on our JSON Server publishers. It is possible that I recently deleted the problematic publisher when I was doing troubleshooting myself, but it would be good to get an extra set of eyes on it. Thanks again!

5

I can't be sure. I've taken a look to those values but I don't know if the DataStore web browser already does some formatting (trim heading and trailing spaces) to the values it shown.

I think that overwriting the server url is a safe change we could attemp. If you actually solved the issue while troubleshooting it won't have any effect and if there's any wrongly configured publisher we'll fix it.

6

@ggalmazor yeah, overwriting the server URL sounds good! Let me know if you need anything from me to proceed.

7

@Jeff_Davids, I've gone through all the publishers at https://console.cloud.google.com/datastore/entities/query?project=s4w-nepal&ns=&kind=opendatakit._json_server3 and none of them has trailing spaces. According to that, the original issue shouldn't happen anymore.

8

Awesome @ggalmazor! I really appreciate your support with this! Enjoy the weekend. :grin:

1 Like