I've been hitting my AppEngine Datastore Read quota and have now enabled
the disable-faster-background-actions checkbox in Aggregate (and also since
cleared out some older publishers which are no longer needed).
···
------
When you set up a new publisher, ODK Aggregate, by default, will attempt to
publish data into that service (Google Spreadsheets, Fusion Tables, your
custom app) *as rapidly as possible*. If you are running using free quota,
this will rapidly deplete the quota.
By checking the disable-faster-background-actions checkbox, ODK Aggregate
will slow down its publishing and will go to sleep after a small number of
rows of data are published. After that, it will only resume publishing data
when incoming submissions arrive and once or twice a day at scheduled times.
------
Can you perhaps clarify/expand on "will go to sleep after a small number of
rows of data are published" portion, as well as "once or twice a day at
scheduled times."
Is "small number" specifically defined? What are the "scheduled times"?
Perhaps a scenario:
A collector captures 100 surveys without an active data link. When they
regain internet access all these forms are uploaded at once.
Will only a few of those 100 (maybe 20?) be published before Aggregate goes
to sleep, another 20 twice that day at the scheduled times, and the rest
only the following day?
A publisher traverses the set of submissions on the server in the order in
which they were marked-as-complete.
If you have 1000 submissions and set up a new "BOTH Upload Existing &
Stream New Submission Data" publisher, it will start with the oldest
marked-as-complete submission and work forward, publishing each, in turn,
into the destination spreadsheet, fusion table, etc.
By default, when incoming form submissions are processed, the server will
attempt to retrieve and publish the 100 oldest not-yet-published
submissions for that form. i.e., it does nothing with the most-recently
submitted data until it has worked through the older ones. There is a
'blackout period' for doing this, such that this is attempted at most every
3 - 6 seconds.
Also by default, at periodic intervals, the server will wake up, scan
through its active publishers, and attempt to publish the 100 oldest
not-yet-published submissions for each of those.
With the checkbox checked, two things happen:
(1) when an incoming submission is processed, it only retrieves the 10
oldest records, not 100.
(2) the blackout period and periodic interval are extended from a few
seconds to 10-15 minutes.
···
On Thu, Aug 20, 2015 at 12:48 AM, Andrew wrote:
Hi team,
I've been hitting my AppEngine Datastore Read quota and have now enabled
the disable-faster-background-actions checkbox in Aggregate (and also since
cleared out some older publishers which are no longer needed).
When you set up a new publisher, ODK Aggregate, by default, will attempt
to publish data into that service (Google Spreadsheets, Fusion Tables, your
custom app) as rapidly as possible. If you are running using free
quota, this will rapidly deplete the quota.
By checking the disable-faster-background-actions checkbox, ODK Aggregate
will slow down its publishing and will go to sleep after a small number of
rows of data are published. After that, it will only resume publishing data
when incoming submissions arrive and once or twice a day at scheduled times.
Can you perhaps clarify/expand on "will go to sleep after a small number
of rows of data are published" portion, as well as "once or twice a day at
scheduled times."
Is "small number" specifically defined? What are the "scheduled times"?
Perhaps a scenario:
A collector captures 100 surveys without an active data link. When they
regain internet access all these forms are uploaded at once.
Will only a few of those 100 (maybe 20?) be published before Aggregate
goes to sleep, another 20 twice that day at the scheduled times, and the
rest only the following day?
···
On Thursday, 20 August 2015 09:48:39 UTC+2, Andrew wrote:
>
> Hi team,
>
> I've been hitting my AppEngine Datastore Read quota and have now enabled
> the disable-faster-background-actions checkbox in Aggregate (and also since
> cleared out some older publishers which are no longer needed).
>
> In an earlier post
> Mitch
> said:
> ------
> When you set up a new publisher, ODK Aggregate, by default, will attempt
> to publish data into that service (Google Spreadsheets, Fusion Tables, your
> custom app) *as rapidly as possible*. If you are running using free
> quota, this will rapidly deplete the quota.
> By checking the disable-faster-background-actions checkbox, ODK Aggregate
> will slow down its publishing and will go to sleep after a small number of
> rows of data are published. After that, it will only resume publishing data
> when incoming submissions arrive and once or twice a day at scheduled times.
> ------
>
> Can you perhaps clarify/expand on "will go to sleep after a small number
> of rows of data are published" portion, as well as "once or twice a day at
> scheduled times."
> Is "small number" specifically defined? What are the "scheduled times"?
>
> Perhaps a scenario:
> A collector captures 100 surveys without an active data link. When they
> regain internet access all these forms are uploaded at once.
> Will only a few of those 100 (maybe 20?) be published before Aggregate
> goes to sleep, another 20 twice that day at the scheduled times, and the
> rest only the following day?
>
>
> I did find a later post
> that
> said:
> With the checkbox checked, publishing can have up to a 10 *minute*
> lag.
> This would indicate that I shouldn't have any undue delays with the
> mentioned scenario - am I correct in my understanding?
>
> Thanks,
> Andrew
>
>