Hi Mitch and Yaw, thanks for your feedback. Some comments on that, and then
some new error info below.
Log - 2015-09-24 - 09h15 - too much contention.txt (11.3 KB)
Log - 2015-09-24 - 13h24.txt (4.73 KB)
···
-------Yaw, turning off the OpenFn mapping initially made a difference to the
usage (as expected), but I've been seeing some other errors (see below)
which might be part of the problem.
Mitch, on the OpenFn usage:
- We're using OpenFn to map the data into SalesForce, where we normally
access it. For the reasons you've mentioned we avoid Aggregate browsing as
far as possible. - OpenFn has up to now run as described earlier (check for new data every
10 mins), but are now testing use of the newer JSON publisher option from
Aggregate, which is obviously a preferred mode of operation. - We also make use of a Google Spreadsheets publisher as there are some
things we can do more easily with pivot tables there than in SalesForce.
Thanks for the tip on the GoogleSpreadsheets notifications - I didn't know
that was available.
I noticed 2 strange things today that I'd appreciate your inputs on:
Problem #1. The publishers on one of our forms have not been publishing
since the beginning of September, even though there have been about 10 new
submissions on this form.
- There are 2 publishers set up, one Google Spreadsheets and one JSON
- Neither of these have published anything since 28 Aug (the next available
data was on 2 Sep) - Both publishers are still showing as active, but the "Published Through"
field shows that they stopped on 1 Sep at 13h41:23 - There are no errors in the logs on 1 Sep
- I created a new Google Spreadsheets publisher for the form, but it didn't
create the Spreadsheet, and the publisher is showing as "not created yet". - I have a second form on the same Aggregate instance which has a Google
Spreadsheets publisher which has been running fine throughout.
Screenshot is attached.
Any ideas what could be causing this behaviour?
Problem #2. There are 2 submissions which are repeatedly being attempted,
but which are failing. A snippet from this morning's log looks like this:
I 2015-09-24 09:16:47.861 200 731 B 390 ms
/aggregateui/servicesadminservice
I 2015-09-24 09:16:51.392 202 0 B 347 ms
/gae/uploadSubmissionsTask?fscUri=uuid%3Adba113c8-4ee2-437f-ae29-492c0d888092
I 2015-09-24 09:16:51.393 202 0 B 224 ms
/gae/uploadSubmissionsTask?fscUri=uuid%3Abb11c9e2-8714-4693-99d4-801ec4a16327
I 2015-09-24 09:16:57.211 200 722 B 126 ms /aggregateui/formservice
I 2015-09-24 09:16:57.610 200 731 B 383 ms
/aggregateui/servicesadminservice
I 2015-09-24 09:17:07.214 200 722 B 157 ms /aggregateui/formservice
I 2015-09-24 09:17:08.053 200 731 B 399 ms
/aggregateui/servicesadminservice
I 2015-09-24 09:17:17.210 200 722 B 128 ms /aggregateui/formservice
I 2015-09-24 09:17:17.737 200 731 B 401 ms
/aggregateui/servicesadminservice
I 2015-09-24 09:17:27.078 200 483 B 170 ms
/aggregateui/servicesadminservice
I 2015-09-24 09:17:27.213 200 722 B 72 ms /aggregateui/formservice
I 2015-09-24 09:17:27.354 202 0 B 392 ms /gae/watchdog
I 2015-09-24 09:17:27.555 200 483 B 74 ms
/aggregateui/servicesadminservice
I 2015-09-24 09:17:30.482 200 731 B 430 ms
/aggregateui/servicesadminservice
I 2015-09-24 09:17:31.503 202 0 B 341 ms
/gae/uploadSubmissionsTask?fscUri=uuid%3Adba113c8-4ee2-437f-ae29-492c0d888092
I 2015-09-24 09:17:31.503 202 0 B 199 ms
/gae/uploadSubmissionsTask?fscUri=uuid%3Abb11c9e2-8714-4693-99d4-801ec4a16327
I 2015-09-24 09:17:37.218 200 722 B 149 ms /aggregateui/formservice
I 2015-09-24 09:17:37.942 200 731 B 350 ms
/aggregateui/servicesadminservice
I 2015-09-24 09:17:47.216 200 722 B 179 ms /aggregateui/formservice
I 2015-09-24 09:17:47.670 200 731 B 343 ms
/aggregateui/servicesadminservice
I 2015-09-24 09:17:57.215 200 722 B 156 ms /aggregateui/formservice
I 2015-09-24 09:17:57.643 200 731 B 389 ms
/aggregateui/servicesadminservice
I 2015-09-24 09:18:07.215 200 722 B 172 ms /aggregateui/formservice
I 2015-09-24 09:18:07.370 202 0 B 619 ms /gae/watchdog
I 2015-09-24 09:18:07.965 200 731 B 375 ms
/aggregateui/servicesadminservice
I 2015-09-24 09:18:11.543 202 0 B 402 ms
/gae/uploadSubmissionsTask?fscUri=uuid%3Adba113c8-4ee2-437f-ae29-492c0d888092
I 2015-09-24 09:18:11.543 202 0 B 337 ms
/gae/uploadSubmissionsTask?fscUri=uuid%3Abb11c9e2-8714-4693-99d4-801ec4a16327
I 2015-09-24 09:37:02.140 202 0 B 763 ms /gae/watchdog
W 2015-09-24 09:37:02.442 202 0 B 522 ms /gae/watchdog
E 2015-09-24 09:37:07.183 202 0 B 1.33 s
/gae/uploadSubmissionsTask?fscUri=uuid%3Abb11c9e2-8714-4693-99d4-801ec4a16327
W 2015-09-24 09:37:07.183 202 0 B 3.39 s
/gae/uploadSubmissionsTask?fscUri=uuid%3Adba113c8-4ee2-437f-ae29-492c0d888092
W 2015-09-24 09:37:07.184 202 0 B 1.12 s
/gae/uploadSubmissionsTask?fscUri=uuid%3Abb11c9e2-8714-4693-99d4-801ec4a16327
I 2015-09-24 09:37:07.184 202 0 B 471 ms
/gae/uploadSubmissionsTask?fscUri=uuid%3Adba113c8-4ee2-437f-ae29-492c0d888092
You will note that these are 2 submissions tasks which are failing and
being resubmitted over and over again.
This is not a quota issue - there was plenty still available when this was
logged.
I have attached 2 log portions showing details. Most of the failures showed
this error:
UNEXPECTED EXCEPTION java.util.ConcurrentModificationException: too
much contention on these datastore entities. please try again.
Followed by:
Unable to acquire lock
2 questions:
a) Any idea what is causing this or how to resolve?
b) Any way to determine which device is submitting these? I can't see
anything except the uuid in the submissions log.
Thanks again for your inputs,
Andrew
On 22 September 2015 at 20:29, Mitch Sundt mitchellsundt@gmail.com wrote:
Using OpenFn for this purpose simply does not make sense.
You are causing ODK Aggregate to fire up every 10 minutes. When it does
this, it reads in every form definition on the server so that it can cache
those definitions in memory. If you have a form with 100 fields, that is at
least 100 reads. If you do that every 10 minutes, that will be 14,400 reads
(624100) every day just to start up the server, and if you have two forms
of this size, that is 28,800 reads every day, again, just to start the
server.If you then actually log in and look at one of the form's submissions tab
(not sure how you are using OpenFn), then each submission page will do at
least 100 reads (to read the first 100 submissions for that form). That's
another 14,400 reads.And looking at a second form submissions tab to check for new data will
cause another 100 reads, minimum. That's another 14,400 reads.At that point, you've burned 57,600 reads.
And you only have 60,000 free reads.
If you have a publisher to Google Spreadsheets, you can configure Google
Spreadsheets to send notifications when it receives a new row. See Tools
/ Notification rules... in the spreadsheet.On Wed, Sep 16, 2015 at 5:14 AM, Yaw Anokwa yanokwa@nafundi.com wrote:
Hi Andrew,
What happens if you disable the OpenFn mappings?
Yaw
Need ODK services? http://nafundi.com provides form design, server
setup, professional support, and software development for ODK.On Wed, Sep 16, 2015 at 7:03 AM, Andrew acawood777@gmail.com wrote:
Hi all,
I've been hitting my datastore read limits for the last while. Have just
turned off one of my unnecessary publishers to see if that helps any,
and
the disable-faster-background-
actions has been ticked for a while already.I've got really low submissions numbers - typically 3 or 4 records per
day.
I do have 2 mappings set up via OpenFn, which check for new submissions
on 2
of the forms every 10 minutes, but I was running fine previously without
these being an issue, so I'm pretty certain they are not the cause.See attached graphs from the dashboard showing 5xx errors every evening
until the quotas are reset at 9am SAST again.I'm beginning to wonder if there was a problem with the initial
Aggregate
install that is causing errors pushing my datastore reads up. But with
devices in the field I want to put-off the haslle of relocating the
forms to
a new Aggregate instance if at all possible.Are there specific errors I should be checking my logs for that can
affect
the number of datastore read operations?
Is there any way to confirm the integrity of the Aggregate instance?
Any other suggestions for tracking down the source of my problems?Thanks,
Andrew--
Post: opendatakit@googlegroups.com
Unsubscribe: opendatakit+unsubscribe@googlegroups.com
Options: http://groups.google.com/group/opendatakit?hl=en
You received this message because you are subscribed to the Google
Groups
"ODK Community" group.
To unsubscribe from this group and stop receiving emails from it, send
an
email to opendatakit+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.--
Post: opendatakit@googlegroups.com
Unsubscribe: opendatakit+unsubscribe@googlegroups.com
Options: http://groups.google.com/group/opendatakit?hl=en
You received this message because you are subscribed to the Google Groups
"ODK Community" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to opendatakit+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.--
Mitch Sundt
Software Engineer
University of Washington
mitchellsundt@gmail.com--
Post: opendatakit@googlegroups.com
Unsubscribe: opendatakit+unsubscribe@googlegroups.com
Options: http://groups.google.com/group/opendatakit?hl=en
You received this message because you are subscribed to a topic in the
Google Groups "ODK Community" group.
To unsubscribe from this topic, visit
https://groups.google.com/d/topic/opendatakit/UvStzcxPNAM/unsubscribe.
To unsubscribe from this group and all its topics, send an email to
opendatakit+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.