Google AppEngine Billing -- expectations

Support
1

Hi everyone,

Google AppEngine is now a production service that charges for usage
exceeding minimal resource limits. If you haven't done so already, go to
your appengine dashboard and accept the new service terms. Those are
reachable from the Billing section of any of your applications via the
adminstration console at https://appengine.google.com/

Please monitor the consumption of resources via the administration
console. If your resource limits are exceeded, Aggregate will report
failures and cease to function.

When Aggregate errors, the first thing to check is that you haven't
exceeded your resource limits. Currently, when a resource limit is
encountered, Aggregate's error messages are difficult to interpret (e.g.,
"Error: Server failure"). We are working to improve our messaging.

An idle Aggregate 1.0 is currently hitting the free-usage resource limit on
"Datastore Read Operations", which is set at 50,000 reads/day.

We will be releasing Aggregate 1.0.1 which will lengthen the time between
/gae/watchdog starts (from 3 minutes to 6 minutes), lengthen the intervals
between web page background-refreshes (from 5 seconds to 10 seconds between
refreshes), and we will be caching more-frequently-used information.
However, with these changes, we still expect that an active Aggregate
instance will exceed the free-usage resource limit. We expect that an
idle or very lightly used Aggregate 1.0.1 will not exceed this
free-resource limit.

From Google's billing help pages,
http://code.google.com/appengine/docs/quotas.html , once you exceed any of
the free resource limits, the minimum charge for an AppEngine application
is $2.10 / week, ($109.20 / year).

The primary resource limit we expect Aggregate to exceed is "Datastore Read
Operations." Additional operations cost $0.07 / 100k reads. If the entire
$2.10 / week goes towards this resource, we expect you will be able to use
Aggregate 1.0.1 fairly actively before exceeding this weekly limit.

An alternative hosting model is Amazon AWS; we are investigating the costs
and options for that platform, but expect AppEngine to be lower-cost for
sporadically- or lightly- used instances (the cross-over point is likely
around $350.00 / year).

----Details-----
Q: what is /gae/watchdog and why is it doing so many reads?

Watchdog is a background activity that runs every 3 minutes (to be
increased to 6 minutes in Aggregate 1.0.1). It ensures that all other
background tasks reach completion in the face of sporadic failures; this
includes publishing to external services, generation of csv and kml file
exports, form deletion, and purging of published submissions. Running
every 3 minutes, it is executed 20 times per hour, or 480 times per day.
For a 50,000 read threshold, this means that watchdog would need to execute
fewer than 105 read operations with each run. Watchdog often has to spin
up a new Aggregate instance -- starting from scratch each time. This means
that each execution must read all the security and form definition records
and then scan all the background task queues for stuck tasks; under these
conditions, it would be extremely challenging to do fewer than 105 read
operations. By increasing the interval to 6 minutes, we can halve the
consumption, but, even then, it is still likely to account for nearly 50%
of the free Datastore Read resource limit.

··· -- Mitch Sundt Software Engineer http://www.OpenDataKit.org University of Washington mitchellsundt@gmail.com
1 Like
2

Mitch,

Thanks for the update.

I have a general question for ODK's vision of use with Appengine.

If ODK data collection with appengine will create instances that are
expected to "exceed the free-usage resource limit", can this system be
used for community level data empowerment? If there will exist a
monthly fees, won't users be more likely to be one time survey
projects or funded research? Obviously the hosting models are still
emerging, but I think the pricing schemes of google strongly suggests
a certain type of usage.

Food for thought for the ODK users out there.. it would be fascinating
to catalogue different types of ODK projects, their goals, and their
funding models.

-AC

··· On Nov 8, 7:20 pm, Mitch Sundt wrote: > Hi everyone, > > Google AppEngine is now a production service that charges for usage > exceeding minimal resource limits. If you haven't done so already, go to > your appengine dashboard and accept the new service terms. Those are > reachable from the Billing section of any of your applications via the > adminstration console athttps://appengine.google.com/ > > Please monitor the consumption of resources via the administration > console. If your resource limits are exceeded, Aggregate will report > failures and cease to function. > > When Aggregate errors, the first thing to check is that you haven't > exceeded your resource limits. Currently, when a resource limit is > encountered, Aggregate's error messages are difficult to interpret (e.g., > "Error: Server failure"). We are working to improve our messaging. > > An idle Aggregate 1.0 is currently hitting the free-usage resource limit on > "Datastore Read Operations", which is set at 50,000 reads/day. > > We will be releasing Aggregate 1.0.1 which will lengthen the time between > /gae/watchdog starts (from 3 minutes to 6 minutes), lengthen the intervals > between web page background-refreshes (from 5 seconds to 10 seconds between > refreshes), and we will be caching more-frequently-used information. > However, with these changes, *we still expect that an active Aggregate > instance will exceed the free-usage resource limit.* We expect that an > idle or very lightly used Aggregate 1.0.1 will not exceed this > free-resource limit. > > From Google's billing help pages,http://code.google.com/appengine/docs/quotas.html, once you exceed any of > the free resource limits, the minimum charge for an AppEngine application > is $2.10 / week, ($109.20 / year). > > The primary resource limit we expect Aggregate to exceed is "Datastore Read > Operations." Additional operations cost $0.07 / 100k reads. If the entire > $2.10 / week goes towards this resource, we expect you will be able to use > Aggregate 1.0.1 fairly actively before exceeding this weekly limit. > > An alternative hosting model is Amazon AWS; we are investigating the costs > and options for that platform, but expect AppEngine to be lower-cost for > sporadically- or lightly- used instances (the cross-over point is likely > around $350.00 / year). > > ----Details----- > Q: what is /gae/watchdog and why is it doing so many reads? > > Watchdog is a background activity that runs every 3 minutes (to be > increased to 6 minutes in Aggregate 1.0.1). It ensures that all other > background tasks reach completion in the face of sporadic failures; this > includes publishing to external services, generation of csv and kml file > exports, form deletion, and purging of published submissions. Running > every 3 minutes, it is executed 20 times per hour, or 480 times per day. > For a 50,000 read threshold, this means that watchdog would need to execute > fewer than 105 read operations with each run. Watchdog often has to spin > up a new Aggregate instance -- starting from scratch each time. This means > that each execution must read all the security and form definition records > and then scan all the background task queues for stuck tasks; under these > conditions, it would be extremely challenging to do fewer than 105 read > operations. By increasing the interval to 6 minutes, we can halve the > consumption, but, even then, it is still likely to account for nearly 50% > of the free Datastore Read resource limit. > > -- > Mitch Sundt > Software Engineerhttp://www.OpenDataKit.org > University of Washington > mitchellsu...@gmail.com
3

I know this is an old issue - but I have been running my Aggregate server for about 2 months now - and suddenly in the past 4 days I have been reaching the Read Datastore Thresholds within about an hour of thier daily refresh. I don't know what is going on - but I don't want to activate billing and find the number of Read Datastore operations spirally out of control.

Is anyone aware of any new developments on this?

··· On Wednesday, November 9, 2011 3:20:04 AM UTC+2, Mitch Sundt wrote: > Hi everyone, > > Google AppEngine is now a production service that charges for usage exceeding minimal resource limits. If you haven't done so already, go to your appengine dashboard and accept the new service terms. Those are reachable from the Billing section of any of your applications via the adminstration console at https://appengine.google.com/ > > > Please monitor the consumption of resources via the administration console. If your resource limits are exceeded, Aggregate will report failures and cease to function. > > When Aggregate errors, the first thing to check is that you haven't exceeded your resource limits. Currently, when a resource limit is encountered, Aggregate's error messages are difficult to interpret (e.g., "Error: Server failure"). We are working to improve our messaging. > > > An idle Aggregate 1.0 is currently hitting the free-usage resource limit on "Datastore Read Operations", which is set at 50,000 reads/day. > > We will be releasing Aggregate 1.0.1 which will lengthen the time between /gae/watchdog starts (from 3 minutes to 6 minutes), lengthen the intervals between web page background-refreshes (from 5 seconds to 10 seconds between refreshes), and we will be caching more-frequently-used information. However, with these changes, we still expect that an active Aggregate instance will exceed the free-usage resource limit. We expect that an idle or very lightly used Aggregate 1.0.1 will not exceed this free-resource limit. > > > From Google's billing help pages, http://code.google.com/appengine/docs/quotas.html , once you exceed any of the free resource limits, the minimum charge for an AppEngine application is $2.10 / week, ($109.20 / year). > > > The primary resource limit we expect Aggregate to exceed is "Datastore Read Operations." Additional operations cost $0.07 / 100k reads. If the entire $2.10 / week goes towards this resource, we expect you will be able to use Aggregate 1.0.1 fairly actively before exceeding this weekly limit. > > > An alternative hosting model is Amazon AWS; we are investigating the costs and options for that platform, but expect AppEngine to be lower-cost for sporadically- or lightly- used instances (the cross-over point is likely around $350.00 / year). > > > ----Details----- > Q: what is /gae/watchdog and why is it doing so many reads? > > Watchdog is a background activity that runs every 3 minutes (to be increased to 6 minutes in Aggregate 1.0.1). It ensures that all other background tasks reach completion in the face of sporadic failures; this includes publishing to external services, generation of csv and kml file exports, form deletion, and purging of published submissions. Running every 3 minutes, it is executed 20 times per hour, or 480 times per day. For a 50,000 read threshold, this means that watchdog would need to execute fewer than 105 read operations with each run. Watchdog often has to spin up a new Aggregate instance -- starting from scratch each time. This means that each execution must read all the security and form definition records and then scan all the background task queues for stuck tasks; under these conditions, it would be extremely challenging to do fewer than 105 read operations. By increasing the interval to 6 minutes, we can halve the consumption, but, even then, it is still likely to account for nearly 50% of the free Datastore Read resource limit. > > > -- > Mitch Sundt > Software Engineer > http://www.OpenDataKit.org > University of Washington > mitche...@gmail.com
4

Hi Mitch,
I am facing the exact same issue as aga.as..@gmail.com
The aggregate instance was working perfectly for the past month, and only in the past 5 days, it almost immediately hits the daily datastore read limit, with most of it from watchdog and upload submission tasks.

I know the uploading to google spreadsheets must be taking up some datastore but it has been working smoothly until now.

Please advise me on the next course on action.
I have followed all the instructions as you mentioned to aga.as..@gmail.com

Thanks!

5

Adam:

your observations are right on. Google has radically changed the
usage limits on AppEngine in the last week. Originally, we had hoped
that GAE would provide a free hosting solution for most light uses and
that only intensive campaigns would require paying for service. It is
now becoming clear that even a small data collection effort will
likely need to pay for the service to the order of USD100 to
USD200/year. Although this is not an outrageous amount, it does
change the model we had hoped for. Unfortunately, all other cloud
solutions that are out there are likely to be at the same cost or
higher. Of course, we always have Aggregate for your own server, but
that raises the bar for technical support and increases costs in
another way. Thus, there is no solution right now that will get us
back to where we were until last week. We are working on some other
possibilities but those are likely some distance away.

Thanks for your patience,
Gaetano

··· On Wed, Nov 9, 2011 at 2:13 PM, Adam Calo wrote: > Mitch, > > Thanks for the update. > > I have a general question for ODK's vision of use with Appengine. > > If ODK data collection with appengine will create instances that are > expected to "exceed the free-usage resource limit", can this system be > used for community level data empowerment? If there will exist a > monthly fees, won't users be more likely to be one time survey > projects or funded research? Obviously the hosting models are still > emerging, but I think the pricing schemes of google strongly suggests > a certain type of usage. > > Food for thought for the ODK users out there.. it would be fascinating > to catalogue different types of ODK projects, their goals, and their > funding models. > > -AC > > On Nov 8, 7:20 pm, Mitch Sundt wrote: >> Hi everyone, >> >> Google AppEngine is now a production service that charges for usage >> exceeding minimal resource limits. If you haven't done so already, go to >> your appengine dashboard and accept the new service terms. Those are >> reachable from the Billing section of any of your applications via the >> adminstration console athttps://appengine.google.com/ >> >> Please monitor the consumption of resources via the administration >> console. If your resource limits are exceeded, Aggregate will report >> failures and cease to function. >> >> When Aggregate errors, the first thing to check is that you haven't >> exceeded your resource limits. Currently, when a resource limit is >> encountered, Aggregate's error messages are difficult to interpret (e.g., >> "Error: Server failure"). We are working to improve our messaging. >> >> An idle Aggregate 1.0 is currently hitting the free-usage resource limit on >> "Datastore Read Operations", which is set at 50,000 reads/day. >> >> We will be releasing Aggregate 1.0.1 which will lengthen the time between >> /gae/watchdog starts (from 3 minutes to 6 minutes), lengthen the intervals >> between web page background-refreshes (from 5 seconds to 10 seconds between >> refreshes), and we will be caching more-frequently-used information. >> However, with these changes, *we still expect that an active Aggregate >> instance will exceed the free-usage resource limit.* We expect that an >> idle or very lightly used Aggregate 1.0.1 will not exceed this >> free-resource limit. >> >> From Google's billing help pages,http://code.google.com/appengine/docs/quotas.html, once you exceed any of >> the free resource limits, the minimum charge for an AppEngine application >> is $2.10 / week, ($109.20 / year). >> >> The primary resource limit we expect Aggregate to exceed is "Datastore Read >> Operations." Additional operations cost $0.07 / 100k reads. If the entire >> $2.10 / week goes towards this resource, we expect you will be able to use >> Aggregate 1.0.1 fairly actively before exceeding this weekly limit. >> >> An alternative hosting model is Amazon AWS; we are investigating the costs >> and options for that platform, but expect AppEngine to be lower-cost for >> sporadically- or lightly- used instances (the cross-over point is likely >> around $350.00 / year). >> >> ----Details----- >> Q: what is /gae/watchdog and why is it doing so many reads? >> >> Watchdog is a background activity that runs every 3 minutes (to be >> increased to 6 minutes in Aggregate 1.0.1). It ensures that all other >> background tasks reach completion in the face of sporadic failures; this >> includes publishing to external services, generation of csv and kml file >> exports, form deletion, and purging of published submissions. Running >> every 3 minutes, it is executed 20 times per hour, or 480 times per day. >> For a 50,000 read threshold, this means that watchdog would need to execute >> fewer than 105 read operations with each run. Watchdog often has to spin >> up a new Aggregate instance -- starting from scratch each time. This means >> that each execution must read all the security and form definition records >> and then scan all the background task queues for stuck tasks; under these >> conditions, it would be extremely challenging to do fewer than 105 read >> operations. By increasing the interval to 6 minutes, we can halve the >> consumption, but, even then, it is still likely to account for nearly 50% >> of the free Datastore Read resource limit. >> >> -- >> Mitch Sundt >> Software Engineerhttp://www.OpenDataKit.org >> University of Washington >> mitchellsu...@gmail.com > > -- > Post: opendatakit@googlegroups.com > Unsubscribe: opendatakit+unsubscribe@googlegroups.com > Options: http://groups.google.com/group/opendatakit?hl=en >
6

Sounds like you might have a datastore corruption problem that is causing
the server to crash and restart.

This can occur if you upgraded to a newer ODK Aggregate server without
following the upgrade steps outlined here:
http://code.google.com/p/opendatakit/wiki/AggregateReleaseNotes

If that might have occurred, it is useful for me to know what older version
of ODK Aggregate you upgraded from.

If you are willing, you can invite me ( mitchellsundt@gmail.com ) to be a
Developer on your system. This is described here:
http://code.google.com/p/opendatakit/wiki/AppEngineTroubleshooting

ALSO:

  • go to your appengine dashboard,
  • click on the "Application Settings" link on the left sidebar beneath the
    "Administration" heading,
  • select "Disable Application" and confirm that action

This will stop your instance from operating (and consuming quota).

Finally, send me (directly -- mitchellsundt@gmail.com ) an e-mail with your
application ID. I will try to resolve the issue as soon as I can (I will
need to wait for the quota to reset, and it might require several resets if
the problem is unexpectedly complex).

··· On Tue, Sep 23, 2014 at 6:33 AM, wrote:

I know this is an old issue - but I have been running my Aggregate server
for about 2 months now - and suddenly in the past 4 days I have been
reaching the Read Datastore Thresholds within about an hour of thier daily
refresh. I don't know what is going on - but I don't want to activate
billing and find the number of Read Datastore operations spirally out of
control.

Is anyone aware of any new developments on this?

On Wednesday, November 9, 2011 3:20:04 AM UTC+2, Mitch Sundt wrote:

Hi everyone,

Google AppEngine is now a production service that charges for usage
exceeding minimal resource limits. If you haven't done so already, go to
your appengine dashboard and accept the new service terms. Those are
reachable from the Billing section of any of your applications via the
adminstration console at https://appengine.google.com/

Please monitor the consumption of resources via the administration
console. If your resource limits are exceeded, Aggregate will report
failures and cease to function.

When Aggregate errors, the first thing to check is that you haven't
exceeded your resource limits. Currently, when a resource limit is
encountered, Aggregate's error messages are difficult to interpret (e.g.,
"Error: Server failure"). We are working to improve our messaging.

An idle Aggregate 1.0 is currently hitting the free-usage resource limit
on "Datastore Read Operations", which is set at 50,000 reads/day.

We will be releasing Aggregate 1.0.1 which will lengthen the time
between /gae/watchdog starts (from 3 minutes to 6 minutes), lengthen the
intervals between web page background-refreshes (from 5 seconds to 10
seconds between refreshes), and we will be caching more-frequently-used
information. However, with these changes, we still expect that an active
Aggregate instance will exceed the free-usage resource limit. We expect
that an idle or very lightly used Aggregate 1.0.1 will not exceed this
free-resource limit.

From Google's billing help pages,
http://code.google.com/appengine/docs/quotas.html , once you exceed any
of the free resource limits, the minimum charge for an AppEngine
application is $2.10 / week, ($109.20 / year).

The primary resource limit we expect Aggregate to exceed is "Datastore
Read Operations." Additional operations cost $0.07 / 100k reads. If the
entire $2.10 / week goes towards this resource, we expect you will be able
to use Aggregate 1.0.1 fairly actively before exceeding this weekly limit.

An alternative hosting model is Amazon AWS; we are investigating the
costs and options for that platform, but expect AppEngine to be lower-cost
for sporadically- or lightly- used instances (the cross-over point is
likely around $350.00 / year).

----Details-----
Q: what is /gae/watchdog and why is it doing so many reads?

Watchdog is a background activity that runs every 3 minutes (to be
increased to 6 minutes in Aggregate 1.0.1). It ensures that all other
background tasks reach completion in the face of sporadic failures; this
includes publishing to external services, generation of csv and kml file
exports, form deletion, and purging of published submissions. Running
every 3 minutes, it is executed 20 times per hour, or 480 times per day.
For a 50,000 read threshold, this means that watchdog would need to execute
fewer than 105 read operations with each run. Watchdog often has to spin
up a new Aggregate instance -- starting from scratch each time. This means
that each execution must read all the security and form definition records
and then scan all the background task queues for stuck tasks; under these
conditions, it would be extremely challenging to do fewer than 105 read
operations. By increasing the interval to 6 minutes, we can halve the
consumption, but, even then, it is still likely to account for nearly 50%
of the free Datastore Read resource limit.

--
Mitch Sundt
Software Engineer
http://www.OpenDataKit.org
University of Washington
mitche...@gmail.com

--
Mitch Sundt
Software Engineer
http://www.OpenDataKit.org http://www.opendatakit.org/
University of Washington
mitchellsundt@gmail.com

7

If the quota is exhausted by the time you start work, you can disable the
application by going to your appengine dashboard, clicking on 'Application
Settings' and then clicking the "Disable Application" button on that page.

Then, once the quota has reset, you can enable the application and then
open a browser, sign in as a Site Admin, and go to the Site Admin /
Preferences page and check the 'Disable faster background actions" checkbox.

This should resolve your issue.

··· On Thu, Oct 16, 2014 at 11:49 AM, wrote:

Hi Mitch,
I am facing the exact same issue as aga.as..@gmail.com
The aggregate instance was working perfectly for the past month, and only
in the past 5 days, it almost immediately hits the daily datastore read
limit, with most of it from watchdog and upload submission tasks.

I know the uploading to google spreadsheets must be taking up some
datastore but it has been working smoothly until now.

Please advise me on the next course on action.
I have followed all the instructions as you mentioned to
aga.as..@gmail.com

Thanks!

--

Post: opendatakit@googlegroups.com
Unsubscribe: opendatakit+unsubscribe@googlegroups.com
Options: http://groups.google.com/group/opendatakit?hl=en

You received this message because you are subscribed to the Google Groups
"ODK Community" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to opendatakit+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--
Mitch Sundt
Software Engineer
University of Washington
mitchellsundt@gmail.com

8

I would also add that this is version 1.0 of Google's pricing.

As a website developer, faced with this free quota threshold and billing
model, my goal would be to design a website that stays below each of these
quota limits for as long as possible, and trips multiple quotas at roughly
the same usage point. This pricing model doesn't allow me to do that -- it
is broken.

Why it is broken is demonstrated by your answer to this question:

Q: How often, when you visit a website, do you author or change anything on
that site by, e.g., posting something, editing or changing data, etc.?

For most of us, we probably view (read) 100x or 1000x more content than we
actually author or change (write).

Yet, with Google's pricing, the free quotas for reads and writes are the
same -- 50,000 / day -- under this billing model, you're expected to author
as much as you read.

And the charge rate is $0.10 for 100k writes and $0.07 for 100k reads --
when all actions relating to writing are factored in, this means that
authoring (writing) content is probably only 4x more expensive than reading
content -- again, this means that websites displaying data are far more
expensive to host on AppEngine than websites that only gather data.

The net effect is to drive developers of today's highly-interactive,
highly-dynamic, "Web 2.0" websites away from AppEngine toward hosting
solutions that have billing models that do not penalize you for generating
dynamic content and having 100x or 1000x as many page views as page updates.

If this billing model stands, then AppEngine will be shunned for all
mainstream web development.

Mitch

··· On Wed, Nov 9, 2011 at 2:24 PM, Gaetano Borriello <gaetano@cs.washington.edu wrote:

Adam:

your observations are right on. Google has radically changed the
usage limits on AppEngine in the last week. Originally, we had hoped
that GAE would provide a free hosting solution for most light uses and
that only intensive campaigns would require paying for service. It is
now becoming clear that even a small data collection effort will
likely need to pay for the service to the order of USD100 to
USD200/year. Although this is not an outrageous amount, it does
change the model we had hoped for. Unfortunately, all other cloud
solutions that are out there are likely to be at the same cost or
higher. Of course, we always have Aggregate for your own server, but
that raises the bar for technical support and increases costs in
another way. Thus, there is no solution right now that will get us
back to where we were until last week. We are working on some other
possibilities but those are likely some distance away.

Thanks for your patience,
Gaetano

On Wed, Nov 9, 2011 at 2:13 PM, Adam Calo adamcalo@gmail.com wrote:

Mitch,

Thanks for the update.

I have a general question for ODK's vision of use with Appengine.

If ODK data collection with appengine will create instances that are
expected to "exceed the free-usage resource limit", can this system be
used for community level data empowerment? If there will exist a
monthly fees, won't users be more likely to be one time survey
projects or funded research? Obviously the hosting models are still
emerging, but I think the pricing schemes of google strongly suggests
a certain type of usage.

Food for thought for the ODK users out there.. it would be fascinating
to catalogue different types of ODK projects, their goals, and their
funding models.

-AC

On Nov 8, 7:20 pm, Mitch Sundt msu...@cs.washington.edu wrote:

Hi everyone,

Google AppEngine is now a production service that charges for usage
exceeding minimal resource limits. If you haven't done so already, go
to
your appengine dashboard and accept the new service terms. Those are
reachable from the Billing section of any of your applications via the
adminstration console athttps://appengine.google.com/

Please monitor the consumption of resources via the administration
console. If your resource limits are exceeded, Aggregate will report
failures and cease to function.

When Aggregate errors, the first thing to check is that you haven't
exceeded your resource limits. Currently, when a resource limit is
encountered, Aggregate's error messages are difficult to interpret
(e.g.,
"Error: Server failure"). We are working to improve our messaging.

An idle Aggregate 1.0 is currently hitting the free-usage resource
limit on
"Datastore Read Operations", which is set at 50,000 reads/day.

We will be releasing Aggregate 1.0.1 which will lengthen the time
between
/gae/watchdog starts (from 3 minutes to 6 minutes), lengthen the
intervals
between web page background-refreshes (from 5 seconds to 10 seconds
between
refreshes), and we will be caching more-frequently-used information.
However, with these changes, we still expect that an active Aggregate
instance will exceed the free-usage resource limit. We expect that an
idle or very lightly used Aggregate 1.0.1 will not exceed this
free-resource limit.

From Google's billing help pages,
http://code.google.com/appengine/docs/quotas.html, once you exceed any of
the free resource limits, the minimum charge for an AppEngine
application
is $2.10 / week, ($109.20 / year).

The primary resource limit we expect Aggregate to exceed is "Datastore
Read
Operations." Additional operations cost $0.07 / 100k reads. If the
entire
$2.10 / week goes towards this resource, we expect you will be able to
use
Aggregate 1.0.1 fairly actively before exceeding this weekly limit.

An alternative hosting model is Amazon AWS; we are investigating the
costs
and options for that platform, but expect AppEngine to be lower-cost for
sporadically- or lightly- used instances (the cross-over point is likely
around $350.00 / year).

----Details-----
Q: what is /gae/watchdog and why is it doing so many reads?

Watchdog is a background activity that runs every 3 minutes (to be
increased to 6 minutes in Aggregate 1.0.1). It ensures that all other
background tasks reach completion in the face of sporadic failures; this
includes publishing to external services, generation of csv and kml file
exports, form deletion, and purging of published submissions. Running
every 3 minutes, it is executed 20 times per hour, or 480 times per day.
For a 50,000 read threshold, this means that watchdog would need to
execute
fewer than 105 read operations with each run. Watchdog often has to
spin
up a new Aggregate instance -- starting from scratch each time. This
means
that each execution must read all the security and form definition
records
and then scan all the background task queues for stuck tasks; under
these
conditions, it would be extremely challenging to do fewer than 105 read
operations. By increasing the interval to 6 minutes, we can halve the
consumption, but, even then, it is still likely to account for nearly
50%
of the free Datastore Read resource limit.

--
Mitch Sundt
Software Engineerhttp://www.OpenDataKit.org
University of Washington
mitchellsu...@gmail.com

--
Post: opendatakit@googlegroups.com
Unsubscribe: opendatakit+unsubscribe@googlegroups.com
Options: http://groups.google.com/group/opendatakit?hl=en

--
Post: opendatakit@googlegroups.com
Unsubscribe: opendatakit+unsubscribe@googlegroups.com
Options: http://groups.google.com/group/opendatakit?hl=en

--
Mitch Sundt
Software Engineer
http://www.OpenDataKit.org
University of Washington
mitchellsundt@gmail.com

9

Hi Mitchell,

Thanks so much for your assistance, and the quick reply! (although
yesterday was a holiday here in South Africa, so I am only checking my mail
again today!)

I have added you as a developer. I would really appreciate the assistance.

Thanks so much!

Miranda Muller

··· On Tue, Sep 23, 2014 at 6:29 PM, Mitchell Sundt wrote:

Sounds like you might have a datastore corruption problem that is causing
the server to crash and restart.

This can occur if you upgraded to a newer ODK Aggregate server without
following the upgrade steps outlined here:
http://code.google.com/p/opendatakit/wiki/AggregateReleaseNotes

If that might have occurred, it is useful for me to know what older
version of ODK Aggregate you upgraded from.

If you are willing, you can invite me ( mitchellsundt@gmail.com ) to be a
Developer on your system. This is described here:
http://code.google.com/p/opendatakit/wiki/AppEngineTroubleshooting

ALSO:

  • go to your appengine dashboard,
  • click on the "Application Settings" link on the left sidebar beneath the
    "Administration" heading,
  • select "Disable Application" and confirm that action

This will stop your instance from operating (and consuming quota).

Finally, send me (directly -- mitchellsundt@gmail.com ) an e-mail with
your application ID. I will try to resolve the issue as soon as I can (I
will need to wait for the quota to reset, and it might require several
resets if the problem is unexpectedly complex).

On Tue, Sep 23, 2014 at 6:33 AM, aga.asmstudy@gmail.com wrote:

I know this is an old issue - but I have been running my Aggregate server
for about 2 months now - and suddenly in the past 4 days I have been
reaching the Read Datastore Thresholds within about an hour of thier daily
refresh. I don't know what is going on - but I don't want to activate
billing and find the number of Read Datastore operations spirally out of
control.

Is anyone aware of any new developments on this?

On Wednesday, November 9, 2011 3:20:04 AM UTC+2, Mitch Sundt wrote:

Hi everyone,

Google AppEngine is now a production service that charges for usage
exceeding minimal resource limits. If you haven't done so already, go to
your appengine dashboard and accept the new service terms. Those are
reachable from the Billing section of any of your applications via the
adminstration console at https://appengine.google.com/

Please monitor the consumption of resources via the administration
console. If your resource limits are exceeded, Aggregate will report
failures and cease to function.

When Aggregate errors, the first thing to check is that you haven't
exceeded your resource limits. Currently, when a resource limit is
encountered, Aggregate's error messages are difficult to interpret (e.g.,
"Error: Server failure"). We are working to improve our messaging.

An idle Aggregate 1.0 is currently hitting the free-usage resource
limit on "Datastore Read Operations", which is set at 50,000 reads/day.

We will be releasing Aggregate 1.0.1 which will lengthen the time
between /gae/watchdog starts (from 3 minutes to 6 minutes), lengthen the
intervals between web page background-refreshes (from 5 seconds to 10
seconds between refreshes), and we will be caching more-frequently-used
information. However, with these changes, we still expect that an active
Aggregate instance will exceed the free-usage resource limit. We expect
that an idle or very lightly used Aggregate 1.0.1 will not exceed this
free-resource limit.

From Google's billing help pages,
http://code.google.com/appengine/docs/quotas.html , once you exceed any
of the free resource limits, the minimum charge for an AppEngine
application is $2.10 / week, ($109.20 / year).

The primary resource limit we expect Aggregate to exceed is "Datastore
Read Operations." Additional operations cost $0.07 / 100k reads. If the
entire $2.10 / week goes towards this resource, we expect you will be able
to use Aggregate 1.0.1 fairly actively before exceeding this weekly limit.

An alternative hosting model is Amazon AWS; we are investigating the
costs and options for that platform, but expect AppEngine to be lower-cost
for sporadically- or lightly- used instances (the cross-over point is
likely around $350.00 / year).

----Details-----
Q: what is /gae/watchdog and why is it doing so many reads?

Watchdog is a background activity that runs every 3 minutes (to be
increased to 6 minutes in Aggregate 1.0.1). It ensures that all other
background tasks reach completion in the face of sporadic failures; this
includes publishing to external services, generation of csv and kml file
exports, form deletion, and purging of published submissions. Running
every 3 minutes, it is executed 20 times per hour, or 480 times per day.
For a 50,000 read threshold, this means that watchdog would need to execute
fewer than 105 read operations with each run. Watchdog often has to spin
up a new Aggregate instance -- starting from scratch each time. This means
that each execution must read all the security and form definition records
and then scan all the background task queues for stuck tasks; under these
conditions, it would be extremely challenging to do fewer than 105 read
operations. By increasing the interval to 6 minutes, we can halve the
consumption, but, even then, it is still likely to account for nearly 50%
of the free Datastore Read resource limit.

--
Mitch Sundt
Software Engineer
http://www.OpenDataKit.org
University of Washington
mitche...@gmail.com

--
Mitch Sundt
Software Engineer
http://www.OpenDataKit.org http://www.opendatakit.org/
University of Washington
mitchellsundt@gmail.com

10

This worked perfectly. Thanks a lot Mitch!

11

Mitch,

Great points - I hope the managers at Google are listening. The algorithms
seem way off.

This may not be related, but I am seeing on an aggregate 1.0 instance of
mine that has been dormant since I did initial testing (i.e. not used but
still up there with exactly two rows in the database from a single form)
that gae_watchdog - which I assume is their automated monitoring tool - i*s
responsible for 50% of my available front-end instance hours in a 14 hr
period *and *36% of my datastore read. * By contrast, I'm running a 9.7
Aggregate instance which has 0 front end requests, and no watchdog activity
and is my primary operational tool so aside from the occasional view - I
have virtually no front end use. (I pass data along to Fusion tables).
Does this make sense?

  • James
··· On Thu, Nov 10, 2011 at 10:05 AM, Mitch Sundt wrote:

I would also add that this is version 1.0 of Google's pricing.

As a website developer, faced with this free quota threshold and billing
model, my goal would be to design a website that stays below each of these
quota limits for as long as possible, and trips multiple quotas at roughly
the same usage point. This pricing model doesn't allow me to do that -- it
is broken.

Why it is broken is demonstrated by your answer to this question:

Q: How often, when you visit a website, do you author or change anything
on that site by, e.g., posting something, editing or changing data, etc.?

For most of us, we probably view (read) 100x or 1000x more content than we
actually author or change (write).

Yet, with Google's pricing, the free quotas for reads and writes are the
same -- 50,000 / day -- under this billing model, you're expected to author
as much as you read.

And the charge rate is $0.10 for 100k writes and $0.07 for 100k reads --
when all actions relating to writing are factored in, this means that
authoring (writing) content is probably only 4x more expensive than reading
content -- again, this means that websites displaying data are far more
expensive to host on AppEngine than websites that only gather data.

The net effect is to drive developers of today's highly-interactive,
highly-dynamic, "Web 2.0" websites away from AppEngine toward hosting
solutions that have billing models that do not penalize you for generating
dynamic content and having 100x or 1000x as many page views as page updates.

If this billing model stands, then AppEngine will be shunned for all
mainstream web development.

Mitch

On Wed, Nov 9, 2011 at 2:24 PM, Gaetano Borriello < gaetano@cs.washington.edu> wrote:

Adam:

your observations are right on. Google has radically changed the
usage limits on AppEngine in the last week. Originally, we had hoped
that GAE would provide a free hosting solution for most light uses and
that only intensive campaigns would require paying for service. It is
now becoming clear that even a small data collection effort will
likely need to pay for the service to the order of USD100 to
USD200/year. Although this is not an outrageous amount, it does
change the model we had hoped for. Unfortunately, all other cloud
solutions that are out there are likely to be at the same cost or
higher. Of course, we always have Aggregate for your own server, but
that raises the bar for technical support and increases costs in
another way. Thus, there is no solution right now that will get us
back to where we were until last week. We are working on some other
possibilities but those are likely some distance away.

Thanks for your patience,
Gaetano

On Wed, Nov 9, 2011 at 2:13 PM, Adam Calo adamcalo@gmail.com wrote:

Mitch,

Thanks for the update.

I have a general question for ODK's vision of use with Appengine.

If ODK data collection with appengine will create instances that are
expected to "exceed the free-usage resource limit", can this system be
used for community level data empowerment? If there will exist a
monthly fees, won't users be more likely to be one time survey
projects or funded research? Obviously the hosting models are still
emerging, but I think the pricing schemes of google strongly suggests
a certain type of usage.

Food for thought for the ODK users out there.. it would be fascinating
to catalogue different types of ODK projects, their goals, and their
funding models.

-AC

On Nov 8, 7:20 pm, Mitch Sundt msu...@cs.washington.edu wrote:

Hi everyone,

Google AppEngine is now a production service that charges for usage
exceeding minimal resource limits. If you haven't done so already, go
to
your appengine dashboard and accept the new service terms. Those are
reachable from the Billing section of any of your applications via the
adminstration console athttps://appengine.google.com/

Please monitor the consumption of resources via the administration
console. If your resource limits are exceeded, Aggregate will report
failures and cease to function.

When Aggregate errors, the first thing to check is that you haven't
exceeded your resource limits. Currently, when a resource limit is
encountered, Aggregate's error messages are difficult to interpret
(e.g.,
"Error: Server failure"). We are working to improve our messaging.

An idle Aggregate 1.0 is currently hitting the free-usage resource
limit on
"Datastore Read Operations", which is set at 50,000 reads/day.

We will be releasing Aggregate 1.0.1 which will lengthen the time
between
/gae/watchdog starts (from 3 minutes to 6 minutes), lengthen the
intervals
between web page background-refreshes (from 5 seconds to 10 seconds
between
refreshes), and we will be caching more-frequently-used information.
However, with these changes, we still expect that an active Aggregate
instance will exceed the free-usage resource limit. We expect that an
idle or very lightly used Aggregate 1.0.1 will not exceed this
free-resource limit.

From Google's billing help pages,
http://code.google.com/appengine/docs/quotas.html, once you exceed any of
the free resource limits, the minimum charge for an AppEngine
application
is $2.10 / week, ($109.20 / year).

The primary resource limit we expect Aggregate to exceed is "Datastore
Read
Operations." Additional operations cost $0.07 / 100k reads. If the
entire
$2.10 / week goes towards this resource, we expect you will be able to
use
Aggregate 1.0.1 fairly actively before exceeding this weekly limit.

An alternative hosting model is Amazon AWS; we are investigating the
costs
and options for that platform, but expect AppEngine to be lower-cost
for
sporadically- or lightly- used instances (the cross-over point is
likely
around $350.00 / year).

----Details-----
Q: what is /gae/watchdog and why is it doing so many reads?

Watchdog is a background activity that runs every 3 minutes (to be
increased to 6 minutes in Aggregate 1.0.1). It ensures that all other
background tasks reach completion in the face of sporadic failures;
this
includes publishing to external services, generation of csv and kml
file
exports, form deletion, and purging of published submissions. Running
every 3 minutes, it is executed 20 times per hour, or 480 times per
day.
For a 50,000 read threshold, this means that watchdog would need to
execute
fewer than 105 read operations with each run. Watchdog often has to
spin
up a new Aggregate instance -- starting from scratch each time. This
means
that each execution must read all the security and form definition
records
and then scan all the background task queues for stuck tasks; under
these
conditions, it would be extremely challenging to do fewer than 105 read
operations. By increasing the interval to 6 minutes, we can halve the
consumption, but, even then, it is still likely to account for nearly
50%
of the free Datastore Read resource limit.

--
Mitch Sundt
Software Engineerhttp://www.OpenDataKit.org
University of Washington
mitchellsu...@gmail.com

--
Post: opendatakit@googlegroups.com
Unsubscribe: opendatakit+unsubscribe@googlegroups.com
Options: http://groups.google.com/group/opendatakit?hl=en

--
Post: opendatakit@googlegroups.com
Unsubscribe: opendatakit+unsubscribe@googlegroups.com
Options: http://groups.google.com/group/opendatakit?hl=en

--
Mitch Sundt
Software Engineer
http://www.OpenDataKit.org
University of Washington
mitchellsundt@gmail.com

--
Post: opendatakit@googlegroups.com
Unsubscribe: opendatakit+unsubscribe@googlegroups.com
Options: http://groups.google.com/group/opendatakit?hl=en

--
James Dailey
skype: jdailey

12

50% is abnormally high. Are there any errors in your log?

/gae/watchdog is part of ODK. It has been set to wake up 480 times a day,
spin up the frontend, check for stalled background jobs (publishing,
primarily, but also data exports, submission purges and form deletions) and
re-trigger them, then go to sleep. We'll be reducing that by 50% with
1.0.1.

Mitch

··· On Thu, Nov 10, 2011 at 3:56 PM, James Dailey wrote:

Mitch,

Great points - I hope the managers at Google are listening. The
algorithms seem way off.

This may not be related, but I am seeing on an aggregate 1.0 instance of
mine that has been dormant since I did initial testing (i.e. not used but
still up there with exactly two rows in the database from a single form)
that gae_watchdog - which I assume is their automated monitoring tool - i*s
responsible for 50% of my available front-end instance hours in a 14 hr
period *and *36% of my datastore read. * By contrast, I'm running a 9.7
Aggregate instance which has 0 front end requests, and no watchdog activity
and is my primary operational tool so aside from the occasional view - I
have virtually no front end use. (I pass data along to Fusion tables).
Does this make sense?

  • James

On Thu, Nov 10, 2011 at 10:05 AM, Mitch Sundt msundt@cs.washington.eduwrote:

I would also add that this is version 1.0 of Google's pricing.

As a website developer, faced with this free quota threshold and billing
model, my goal would be to design a website that stays below each of these
quota limits for as long as possible, and trips multiple quotas at roughly
the same usage point. This pricing model doesn't allow me to do that -- it
is broken.

Why it is broken is demonstrated by your answer to this question:

Q: How often, when you visit a website, do you author or change anything
on that site by, e.g., posting something, editing or changing data, etc.?

For most of us, we probably view (read) 100x or 1000x more content than
we actually author or change (write).

Yet, with Google's pricing, the free quotas for reads and writes are the
same -- 50,000 / day -- under this billing model, you're expected to author
as much as you read.

And the charge rate is $0.10 for 100k writes and $0.07 for 100k reads --
when all actions relating to writing are factored in, this means that
authoring (writing) content is probably only 4x more expensive than reading
content -- again, this means that websites displaying data are far more
expensive to host on AppEngine than websites that only gather data.

The net effect is to drive developers of today's highly-interactive,
highly-dynamic, "Web 2.0" websites away from AppEngine toward hosting
solutions that have billing models that do not penalize you for generating
dynamic content and having 100x or 1000x as many page views as page updates.

If this billing model stands, then AppEngine will be shunned for all
mainstream web development.

Mitch

On Wed, Nov 9, 2011 at 2:24 PM, Gaetano Borriello < gaetano@cs.washington.edu> wrote:

Adam:

your observations are right on. Google has radically changed the
usage limits on AppEngine in the last week. Originally, we had hoped
that GAE would provide a free hosting solution for most light uses and
that only intensive campaigns would require paying for service. It is
now becoming clear that even a small data collection effort will
likely need to pay for the service to the order of USD100 to
USD200/year. Although this is not an outrageous amount, it does
change the model we had hoped for. Unfortunately, all other cloud
solutions that are out there are likely to be at the same cost or
higher. Of course, we always have Aggregate for your own server, but
that raises the bar for technical support and increases costs in
another way. Thus, there is no solution right now that will get us
back to where we were until last week. We are working on some other
possibilities but those are likely some distance away.

Thanks for your patience,
Gaetano

On Wed, Nov 9, 2011 at 2:13 PM, Adam Calo adamcalo@gmail.com wrote:

Mitch,

Thanks for the update.

I have a general question for ODK's vision of use with Appengine.

If ODK data collection with appengine will create instances that are
expected to "exceed the free-usage resource limit", can this system be
used for community level data empowerment? If there will exist a
monthly fees, won't users be more likely to be one time survey
projects or funded research? Obviously the hosting models are still
emerging, but I think the pricing schemes of google strongly suggests
a certain type of usage.

Food for thought for the ODK users out there.. it would be fascinating
to catalogue different types of ODK projects, their goals, and their
funding models.

-AC

On Nov 8, 7:20 pm, Mitch Sundt msu...@cs.washington.edu wrote:

Hi everyone,

Google AppEngine is now a production service that charges for usage
exceeding minimal resource limits. If you haven't done so already,
go to
your appengine dashboard and accept the new service terms. Those are
reachable from the Billing section of any of your applications via the
adminstration console athttps://appengine.google.com/

Please monitor the consumption of resources via the administration
console. If your resource limits are exceeded, Aggregate will report
failures and cease to function.

When Aggregate errors, the first thing to check is that you haven't
exceeded your resource limits. Currently, when a resource limit is
encountered, Aggregate's error messages are difficult to interpret
(e.g.,
"Error: Server failure"). We are working to improve our messaging.

An idle Aggregate 1.0 is currently hitting the free-usage resource
limit on
"Datastore Read Operations", which is set at 50,000 reads/day.

We will be releasing Aggregate 1.0.1 which will lengthen the time
between
/gae/watchdog starts (from 3 minutes to 6 minutes), lengthen the
intervals
between web page background-refreshes (from 5 seconds to 10 seconds
between
refreshes), and we will be caching more-frequently-used information.
However, with these changes, we still expect that an active Aggregate
instance will exceed the free-usage resource limit. We expect that
an
idle or very lightly used Aggregate 1.0.1 will not exceed this
free-resource limit.

From Google's billing help pages,
http://code.google.com/appengine/docs/quotas.html, once you exceed any
of
the free resource limits, the minimum charge for an AppEngine
application
is $2.10 / week, ($109.20 / year).

The primary resource limit we expect Aggregate to exceed is
"Datastore Read
Operations." Additional operations cost $0.07 / 100k reads. If the
entire
$2.10 / week goes towards this resource, we expect you will be able
to use
Aggregate 1.0.1 fairly actively before exceeding this weekly limit.

An alternative hosting model is Amazon AWS; we are investigating the
costs
and options for that platform, but expect AppEngine to be lower-cost
for
sporadically- or lightly- used instances (the cross-over point is
likely
around $350.00 / year).

----Details-----
Q: what is /gae/watchdog and why is it doing so many reads?

Watchdog is a background activity that runs every 3 minutes (to be
increased to 6 minutes in Aggregate 1.0.1). It ensures that all other
background tasks reach completion in the face of sporadic failures;
this
includes publishing to external services, generation of csv and kml
file
exports, form deletion, and purging of published submissions. Running
every 3 minutes, it is executed 20 times per hour, or 480 times per
day.
For a 50,000 read threshold, this means that watchdog would need to
execute
fewer than 105 read operations with each run. Watchdog often has to
spin
up a new Aggregate instance -- starting from scratch each time. This
means
that each execution must read all the security and form definition
records
and then scan all the background task queues for stuck tasks; under
these
conditions, it would be extremely challenging to do fewer than 105
read
operations. By increasing the interval to 6 minutes, we can halve the
consumption, but, even then, it is still likely to account for nearly
50%
of the free Datastore Read resource limit.

--
Mitch Sundt
Software Engineerhttp://www.OpenDataKit.org
University of Washington
mitchellsu...@gmail.com

--
Post: opendatakit@googlegroups.com
Unsubscribe: opendatakit+unsubscribe@googlegroups.com
Options: http://groups.google.com/group/opendatakit?hl=en

--
Post: opendatakit@googlegroups.com
Unsubscribe: opendatakit+unsubscribe@googlegroups.com
Options: http://groups.google.com/group/opendatakit?hl=en

--
Mitch Sundt
Software Engineer
http://www.OpenDataKit.org
University of Washington
mitchellsundt@gmail.com

--
Post: opendatakit@googlegroups.com
Unsubscribe: opendatakit+unsubscribe@googlegroups.com
Options: http://groups.google.com/group/opendatakit?hl=en

--
James Dailey
skype: jdailey

--
Post: opendatakit@googlegroups.com
Unsubscribe: opendatakit+unsubscribe@googlegroups.com
Options: http://groups.google.com/group/opendatakit?hl=en

--
Mitch Sundt
Software Engineer
http://www.OpenDataKit.org
University of Washington
mitchellsundt@gmail.com

13

Actually, I just calculated this out -- 1.0 fires off watchdog every 3
minutes. It seems AppEngine keeps an instance alive for about 3 minutes.
So watchdog would consume about 24 hours of frontend usage, or about 86% of
quota.

With Aggregate 1.0.1, this would drop to 43% of quota, since we fire off a
watchdog every 6 minutes.

Mitch

··· On Thu, Nov 10, 2011 at 6:53 PM, Mitch Sundt wrote:

50% is abnormally high. Are there any errors in your log?

/gae/watchdog is part of ODK. It has been set to wake up 480 times a day,
spin up the frontend, check for stalled background jobs (publishing,
primarily, but also data exports, submission purges and form deletions) and
re-trigger them, then go to sleep. We'll be reducing that by 50% with
1.0.1.

Mitch

On Thu, Nov 10, 2011 at 3:56 PM, James Dailey jamespdailey@gmail.comwrote:

Mitch,

Great points - I hope the managers at Google are listening. The
algorithms seem way off.

This may not be related, but I am seeing on an aggregate 1.0 instance of
mine that has been dormant since I did initial testing (i.e. not used but
still up there with exactly two rows in the database from a single form)
that gae_watchdog - which I assume is their automated monitoring tool - i
*s responsible for 50% of my available front-end instance hours in a 14
hr period *and *36% of my datastore read. * By contrast, I'm running a
9.7 Aggregate instance which has 0 front end requests, and no watchdog
activity and is my primary operational tool so aside from the occasional
view - I have virtually no front end use. (I pass data along to Fusion
tables). Does this make sense?

  • James

On Thu, Nov 10, 2011 at 10:05 AM, Mitch Sundt msundt@cs.washington.eduwrote:

I would also add that this is version 1.0 of Google's pricing.

As a website developer, faced with this free quota threshold and billing
model, my goal would be to design a website that stays below each of these
quota limits for as long as possible, and trips multiple quotas at roughly
the same usage point. This pricing model doesn't allow me to do that -- it
is broken.

Why it is broken is demonstrated by your answer to this question:

Q: How often, when you visit a website, do you author or change anything
on that site by, e.g., posting something, editing or changing data, etc.?

For most of us, we probably view (read) 100x or 1000x more content than
we actually author or change (write).

Yet, with Google's pricing, the free quotas for reads and writes are the
same -- 50,000 / day -- under this billing model, you're expected to author
as much as you read.

And the charge rate is $0.10 for 100k writes and $0.07 for 100k reads --
when all actions relating to writing are factored in, this means that
authoring (writing) content is probably only 4x more expensive than reading
content -- again, this means that websites displaying data are far more
expensive to host on AppEngine than websites that only gather data.

The net effect is to drive developers of today's highly-interactive,
highly-dynamic, "Web 2.0" websites away from AppEngine toward hosting
solutions that have billing models that do not penalize you for generating
dynamic content and having 100x or 1000x as many page views as page updates.

If this billing model stands, then AppEngine will be shunned for all
mainstream web development.

Mitch

On Wed, Nov 9, 2011 at 2:24 PM, Gaetano Borriello < gaetano@cs.washington.edu> wrote:

Adam:

your observations are right on. Google has radically changed the
usage limits on AppEngine in the last week. Originally, we had hoped
that GAE would provide a free hosting solution for most light uses and
that only intensive campaigns would require paying for service. It is
now becoming clear that even a small data collection effort will
likely need to pay for the service to the order of USD100 to
USD200/year. Although this is not an outrageous amount, it does
change the model we had hoped for. Unfortunately, all other cloud
solutions that are out there are likely to be at the same cost or
higher. Of course, we always have Aggregate for your own server, but
that raises the bar for technical support and increases costs in
another way. Thus, there is no solution right now that will get us
back to where we were until last week. We are working on some other
possibilities but those are likely some distance away.

Thanks for your patience,
Gaetano

On Wed, Nov 9, 2011 at 2:13 PM, Adam Calo adamcalo@gmail.com wrote:

Mitch,

Thanks for the update.

I have a general question for ODK's vision of use with Appengine.

If ODK data collection with appengine will create instances that are
expected to "exceed the free-usage resource limit", can this system be
used for community level data empowerment? If there will exist a
monthly fees, won't users be more likely to be one time survey
projects or funded research? Obviously the hosting models are still
emerging, but I think the pricing schemes of google strongly suggests
a certain type of usage.

Food for thought for the ODK users out there.. it would be fascinating
to catalogue different types of ODK projects, their goals, and their
funding models.

-AC

On Nov 8, 7:20 pm, Mitch Sundt msu...@cs.washington.edu wrote:

Hi everyone,

Google AppEngine is now a production service that charges for usage
exceeding minimal resource limits. If you haven't done so already,
go to
your appengine dashboard and accept the new service terms. Those are
reachable from the Billing section of any of your applications via
the
adminstration console athttps://appengine.google.com/

Please monitor the consumption of resources via the administration
console. If your resource limits are exceeded, Aggregate will report
failures and cease to function.

When Aggregate errors, the first thing to check is that you haven't
exceeded your resource limits. Currently, when a resource limit is
encountered, Aggregate's error messages are difficult to interpret
(e.g.,
"Error: Server failure"). We are working to improve our messaging.

An idle Aggregate 1.0 is currently hitting the free-usage resource
limit on
"Datastore Read Operations", which is set at 50,000 reads/day.

We will be releasing Aggregate 1.0.1 which will lengthen the time
between
/gae/watchdog starts (from 3 minutes to 6 minutes), lengthen the
intervals
between web page background-refreshes (from 5 seconds to 10 seconds
between
refreshes), and we will be caching more-frequently-used information.
However, with these changes, we still expect that an active
Aggregate
instance will exceed the free-usage resource limit. We expect that
an
idle or very lightly used Aggregate 1.0.1 will not exceed this
free-resource limit.

From Google's billing help pages,
http://code.google.com/appengine/docs/quotas.html, once you exceed any
of
the free resource limits, the minimum charge for an AppEngine
application
is $2.10 / week, ($109.20 / year).

The primary resource limit we expect Aggregate to exceed is
"Datastore Read
Operations." Additional operations cost $0.07 / 100k reads. If the
entire
$2.10 / week goes towards this resource, we expect you will be able
to use
Aggregate 1.0.1 fairly actively before exceeding this weekly limit.

An alternative hosting model is Amazon AWS; we are investigating the
costs
and options for that platform, but expect AppEngine to be lower-cost
for
sporadically- or lightly- used instances (the cross-over point is
likely
around $350.00 / year).

----Details-----
Q: what is /gae/watchdog and why is it doing so many reads?

Watchdog is a background activity that runs every 3 minutes (to be
increased to 6 minutes in Aggregate 1.0.1). It ensures that all other
background tasks reach completion in the face of sporadic failures;
this
includes publishing to external services, generation of csv and kml
file
exports, form deletion, and purging of published submissions.
Running
every 3 minutes, it is executed 20 times per hour, or 480 times per
day.
For a 50,000 read threshold, this means that watchdog would need to
execute
fewer than 105 read operations with each run. Watchdog often has to
spin
up a new Aggregate instance -- starting from scratch each time.
This means
that each execution must read all the security and form definition
records
and then scan all the background task queues for stuck tasks; under
these
conditions, it would be extremely challenging to do fewer than 105
read
operations. By increasing the interval to 6 minutes, we can halve
the
consumption, but, even then, it is still likely to account for
nearly 50%
of the free Datastore Read resource limit.

--
Mitch Sundt
Software Engineerhttp://www.OpenDataKit.org
University of Washington
mitchellsu...@gmail.com

--
Post: opendatakit@googlegroups.com
Unsubscribe: opendatakit+unsubscribe@googlegroups.com
Options: http://groups.google.com/group/opendatakit?hl=en

--
Post: opendatakit@googlegroups.com
Unsubscribe: opendatakit+unsubscribe@googlegroups.com
Options: http://groups.google.com/group/opendatakit?hl=en

--
Mitch Sundt
Software Engineer
http://www.OpenDataKit.org
University of Washington
mitchellsundt@gmail.com

--
Post: opendatakit@googlegroups.com
Unsubscribe: opendatakit+unsubscribe@googlegroups.com
Options: http://groups.google.com/group/opendatakit?hl=en

--
James Dailey
skype: jdailey

--
Post: opendatakit@googlegroups.com
Unsubscribe: opendatakit+unsubscribe@googlegroups.com
Options: http://groups.google.com/group/opendatakit?hl=en

--
Mitch Sundt
Software Engineer
http://www.OpenDataKit.org
University of Washington
mitchellsundt@gmail.com

--
Mitch Sundt
Software Engineer
http://www.OpenDataKit.org
University of Washington
mitchellsundt@gmail.com

14

Hi Mitch, im working with odk and app engine for a month and suddenly i find a problem with my watchdog.
I need a help to setup the watchdog and fix the problem.

Can you help me?

"sorry with the english"

i´m nedding a help to setup my watchdog, to working in 6 minutes.

The watchdog is consuming all of my data of my free app engine.

15

This is a very old thread.

The background instance runs whenever you:

  1. create publishers to publish all existing records to some downstream server.

  2. delete a form and all of its submissions.

  3. choose Export to CSV or KML.

Any of these activities can quickly consume your quota.

If you want to run these actions slower, and preserve your quota for submission and interactive website interactions, you can go to the Site Admin / Preferences page and check the
"Disable faster background actions (exports, publishing, form deletion) (slows quota usage on Google AppEngine)" checkbox.

And, rather than use the server's export to CSV feature, we recommend using ODK Briefcase and then generating the CSV locally on your computer.

1 Like