I have set up Aggregate on Google App Engine. I added two submissions and then pushed them to google fusion tables for testing purposes. Now, when I try to visit Aggregate it says that I am over my quota or having an error. I looked at my quotas and the Backend Instance Hours is over the quota.
Does anyone know what could be causing me to go over the quota?
Backend Instance Hours
9.02 of 9 Instance Hours
100%
Standard
Here is what I found
at https://opendatakit.org/use/aggregate/deployment-planning/
App Engine Usage Fees¶
https://opendatakit.org/use/aggregate/deployment-planning/#App_Engine_Usage_Fees
On App Engine, the primary consumption is Datastore Reads.
To minimize App Engine usage fees, users should restrict access to the ODK
Aggregate website and NOTkeep browser windows open on the submissions tab.
That tab is refreshed every 6 seconds if the user is actively doing work,
or slightly less frequently if they are not. At the default display of 100
submissions (a minimum of 100 Reads), that can quickly add up. Keep in mind
that every select-one or select-multiple question incurs an additional Read
(one for each value stored). And images incur a minimum of 10 Reads. Every
200 questions in your survey costs an additional Read, and each repeat
group also costs a Read for each filled-in repeat in that group.
What version of ODK Aggregate are you using?
Here is what I found at
https://opendatakit.org/use/aggregate/deployment-planning/App Engine Usage Fees¶
https://opendatakit.org/use/aggregate/deployment-planning/#App_Engine_Usage_FeesOn App Engine, the primary consumption is Datastore Reads.
To minimize App Engine usage fees, users should restrict access to the ODK
Aggregate website and NOTkeep browser windows open on the submissions tab.
That tab is refreshed every 6 seconds if the user is actively doing work,
or slightly less frequently if they are not. At the default display of 100
submissions (a minimum of 100 Reads), that can quickly add up. Keep in mind
that every select-one or select-multiple question incurs an additional Read
(one for each value stored). And images incur a minimum of 10 Reads. Every
200 questions in your survey costs an additional Read, and each repeat
group also costs a Read for each filled-in repeat in that group.On Saturday, May 7, 2016 at 5:28:49 PM UTC-6, Jefferson Francisco wrote:
I have set up Aggregate on Google App Engine. I added two submissions and
then pushed them to google fusion tables for testing purposes. Now, when I
try to visit Aggregate it says that I am over my quota or having an error.
I looked at my quotas and the Backend Instance Hours is over the quota.Does anyone know what could be causing me to go over the quota?
Backend Instance Hours
9.02 of 9 Instance Hours
100%
Standard--
Post: opendatakit@googlegroups.com
Unsubscribe: opendatakit+unsubscribe@googlegroups.com
Options: http://groups.google.com/group/opendatakit?hl=en
You received this message because you are subscribed to the Google Groups
"ODK Community" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to opendatakit+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
--
Mitch Sundt
Software Engineer
University of Washington
mitchellsundt@gmail.com
1.4.8
Is this a clean install or an upgrade from an earlier version?
If it is an upgrade, beginning with ODK Aggregate v1.4.8, we no longer use
backends.
So the error would suggest that there is a rogue backend on your system.
I would recommend:
(0) Verify that you are running ODK Aggregate v1.4.8 by logging in with
Site Admin privileges and viewing the Site Admin / Preferences screen.
(1) close all browser windows to your ODK Aggregate server.
(2) go to the AppEngine instances console:
(3) Select your project / application ID in the top right of the screen.
(4) Then for the "default" and "background" instances, choose Shutdown to
stop them.
(5) If you weren't running v1.4.8, upgrade to v1.4.8
The quota limits are reset daily, so this should clear fairly quickly.
1.4.8
On Sunday, May 8, 2016 at 11:48:13 AM UTC-6, Mitch Sundt wrote:
What version of ODK Aggregate are you using?
On Sat, May 7, 2016 at 4:41 PM, Jefferson Francisco < jeffersont...@gmail.com> wrote:
Here is what I found at
https://opendatakit.org/use/aggregate/deployment-planning/App Engine Usage Fees¶
https://opendatakit.org/use/aggregate/deployment-planning/#App_Engine_Usage_FeesOn App Engine, the primary consumption is Datastore Reads.
To minimize App Engine usage fees, users should restrict access to the
ODK Aggregate website and NOTkeep browser windows open on the submissions
tab. That tab is refreshed every 6 seconds if the user is actively doing
work, or slightly less frequently if they are not. At the default display
of 100 submissions (a minimum of 100 Reads), that can quickly add up. Keep
in mind that every select-one or select-multiple question incurs an
additional Read (one for each value stored). And images incur a minimum of
10 Reads. Every 200 questions in your survey costs an additional Read, and
each repeat group also costs a Read for each filled-in repeat in that group.On Saturday, May 7, 2016 at 5:28:49 PM UTC-6, Jefferson Francisco wrote:
I have set up Aggregate on Google App Engine. I added two submissions
and then pushed them to google fusion tables for testing purposes. Now,
when I try to visit Aggregate it says that I am over my quota or having an
error. I looked at my quotas and the Backend Instance Hours is over the
quota.Does anyone know what could be causing me to go over the quota?
Backend Instance Hours
9.02 of 9 Instance Hours
100%
Standard--
Post: opend...@googlegroups.com
Unsubscribe: opendatakit...@googlegroups.com
Options: http://groups.google.com/group/opendatakit?hl=en
You received this message because you are subscribed to the Google
Groups "ODK Community" group.
To unsubscribe from this group and stop receiving emails from it, send
an email to opendatakit...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.--
Mitch Sundt
Software Engineer
University of Washington
mitche...@gmail.com--
Post: opendatakit@googlegroups.com
Unsubscribe: opendatakit+unsubscribe@googlegroups.com
Options: http://groups.google.com/group/opendatakit?hl=en
You received this message because you are subscribed to the Google Groups
"ODK Community" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to opendatakit+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
--
Mitch Sundt
Software Engineer
University of Washington
mitchellsundt@gmail.com
It is a clean install. I verified that it is v1.4.8
Today, I logged into Aggregate and updated my api certificate and keys.
Then I published the 2 submissions. I have probably spent a total of 30
minutes on Aggregate today. I also checked the first option on the bottom
of the Site admin / preferences page to slow down background jobs.
Here are my quota stats for today:
Frontend Instance Hours 32%
Datastore Read Operations 11%
Datastore Write Operations 1%
Does this look like normal quota usage?
The datastore reads and writes look correct -- it is about a 10x difference
in usage of reads vs. writes.
The read quota limit is what generally forces people to set up billing (the
limit is at 60,000, and then the first 1,000,000 cost $0.60).
Google recently changed their billing and quota accounting, and we recently
changed away from "Backend" servers to "Modules".
As a result, I don't have a good sense of whether that made everything cost
more or not.
The Frontend instance hours will be larger than the older "Backend" system,
but I don't have a sense of whether this is anomalous.
To minimize usage, all form definitions are read once at the time the
server starts, and re-read every few minutes thereafter.
Checking the checkbox will definitely reduce that cost, but will insert
delays in the publishing of submitted data to Fusion Tables. Closing any
browser windows to your server will also minimize this. If you have
multiple browser windows open, this can cause Google to spin up 2 or more
copies of the server to handle the refresh-requests that the browser makes,
and the running times of each copy counts toward the frontend instance
hours, as does the time the publisher to Fusion Tables runs on the
background module.
Other costs depend upon how complex your form is (because that complexity
translates into read requests)
If it has a lot of repeat groups, and a lot of multiple-choice questions,
then it will cost more to process. As do forms with 100's of questions
(whether answered or not -- it is the total number of potential questions
that impacts costs).
Large or high-resolution video clips or high-resolution images are also
expensive (and, again, storage is cheap, it is costly because of the read
requests)
On Sun, May 8, 2016 at 12:13 PM, Jefferson Francisco < jeffersont.francisco@gmail.com> wrote:
It is a clean install. I verified that it is v1.4.8
Today, I logged into Aggregate and updated my api certificate and keys.
Then I published the 2 submissions. I have probably spent a total of 30
minutes on Aggregate today. I also checked the first option on the bottom
of the Site admin / preferences page to slow down background jobs.Here are my quota stats for today:
Frontend Instance Hours 32%
Datastore Read Operations 11%
Datastore Write Operations 1%Does this look like normal quota usage?
On Sunday, May 8, 2016 at 12:27:13 PM UTC-6, Mitch Sundt wrote:
Is this a clean install or an upgrade from an earlier version?
If it is an upgrade, beginning with ODK Aggregate v1.4.8, we no longer
use backends.
So the error would suggest that there is a rogue backend on your system.I would recommend:
(0) Verify that you are running ODK Aggregate v1.4.8 by logging in with
Site Admin privileges and viewing the Site Admin / Preferences screen.(1) close all browser windows to your ODK Aggregate server.
(2) go to the AppEngine instances console:
https://console.cloud.google.com/appengine/instances
(3) Select your project / application ID in the top right of the screen.
(4) Then for the "default" and "background" instances, choose Shutdown to
stop them.(5) If you weren't running v1.4.8, upgrade to v1.4.8
The quota limits are reset daily, so this should clear fairly quickly.
On Sun, May 8, 2016 at 10:54 AM, Jefferson Francisco < jeffersont...@gmail.com> wrote:
1.4.8
On Sunday, May 8, 2016 at 11:48:13 AM UTC-6, Mitch Sundt wrote:
What version of ODK Aggregate are you using?
On Sat, May 7, 2016 at 4:41 PM, Jefferson Francisco < jeffersont...@gmail.com> wrote:
Here is what I found at
https://opendatakit.org/use/aggregate/deployment-planning/App Engine Usage Fees¶
https://opendatakit.org/use/aggregate/deployment-planning/#App_Engine_Usage_FeesOn App Engine, the primary consumption is Datastore Reads.
To minimize App Engine usage fees, users should restrict access to the
ODK Aggregate website and NOTkeep browser windows open on the submissions
tab. That tab is refreshed every 6 seconds if the user is actively doing
work, or slightly less frequently if they are not. At the default display
of 100 submissions (a minimum of 100 Reads), that can quickly add up. Keep
in mind that every select-one or select-multiple question incurs an
additional Read (one for each value stored). And images incur a minimum of
10 Reads. Every 200 questions in your survey costs an additional Read, and
each repeat group also costs a Read for each filled-in repeat in that group.On Saturday, May 7, 2016 at 5:28:49 PM UTC-6, Jefferson Francisco wrote:
I have set up Aggregate on Google App Engine. I added two submissions
and then pushed them to google fusion tables for testing purposes. Now,
when I try to visit Aggregate it says that I am over my quota or having an
error. I looked at my quotas and the Backend Instance Hours is over the
quota.Does anyone know what could be causing me to go over the quota?
Backend Instance Hours
9.02 of 9 Instance Hours
100%
Standard--
Post: opend...@googlegroups.com
Unsubscribe: opendatakit...@googlegroups.com
Options: http://groups.google.com/group/opendatakit?hl=en
You received this message because you are subscribed to the Google
Groups "ODK Community" group.
To unsubscribe from this group and stop receiving emails from it, send
an email to opendatakit...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.--
Mitch Sundt
Software Engineer
University of Washington
mitche...@gmail.com--
Post: opend...@googlegroups.com
Unsubscribe: opendatakit...@googlegroups.com
Options: http://groups.google.com/group/opendatakit?hl=en
You received this message because you are subscribed to the Google
Groups "ODK Community" group.
To unsubscribe from this group and stop receiving emails from it, send
an email to opendatakit...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.--
Mitch Sundt
Software Engineer
University of Washington
mitche...@gmail.com--
Post: opendatakit@googlegroups.com
Unsubscribe: opendatakit+unsubscribe@googlegroups.com
Options: http://groups.google.com/group/opendatakit?hl=en
You received this message because you are subscribed to the Google Groups
"ODK Community" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to opendatakit+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
--
Mitch Sundt
Software Engineer
University of Washington
mitchellsundt@gmail.com
I really appreciate the explanation. We are setting up Aggregate to monitor
projects with about 50,000 beneficiaries. So, we are exploring a lot of
options to keep the cost as low as possible. I am going to experiment to
see the costs involved with images. In the grand scheme of things it
probably won't be that expensive, but I will share any interesting findings
about cost that I find.
I have just put the ODK 1.4.9 installers up on the site -- download and
upgrade to 1.4.9.
https://opendatakit.org/downloads/download-category/aggregate/
I just realized that the out-of-quota is probably due to an Oauth2 bug that
was created in 1.4.8 and is now fixed in 1.4.9.
I really appreciate the explanation. We are setting up Aggregate to
monitor projects with about 50,000 beneficiaries. So, we are exploring a
lot of options to keep the cost as low as possible. I am going to
experiment to see the costs involved with images. In the grand scheme of
things it probably won't be that expensive, but I will share any
interesting findings about cost that I find.On Sunday, May 8, 2016 at 1:46:06 PM UTC-6, Mitch Sundt wrote:
The datastore reads and writes look correct -- it is about a 10x
difference in usage of reads vs. writes.
The read quota limit is what generally forces people to set up billing
(the limit is at 60,000, and then the first 1,000,000 cost $0.60).Google recently changed their billing and quota accounting, and we
recently changed away from "Backend" servers to "Modules".
As a result, I don't have a good sense of whether that made everything
cost more or not.The Frontend instance hours will be larger than the older "Backend"
system, but I don't have a sense of whether this is anomalous.=======
The most costly operations on the system are deleting and uploading a
form definition. The datastore read usages are reasonable for having done
that, and having set up publishing.To minimize usage, all form definitions are read once at the time the
server starts, and re-read every few minutes thereafter.Checking the checkbox will definitely reduce that cost, but will insert
delays in the publishing of submitted data to Fusion Tables. Closing any
browser windows to your server will also minimize this. If you have
multiple browser windows open, this can cause Google to spin up 2 or more
copies of the server to handle the refresh-requests that the browser makes,
and the running times of each copy counts toward the frontend instance
hours, as does the time the publisher to Fusion Tables runs on the
background module.Other costs depend upon how complex your form is (because that complexity
translates into read requests)If it has a lot of repeat groups, and a lot of multiple-choice questions,
then it will cost more to process. As do forms with 100's of questions
(whether answered or not -- it is the total number of potential questions
that impacts costs).Large or high-resolution video clips or high-resolution images are also
expensive (and, again, storage is cheap, it is costly because of the read
requests)On Sun, May 8, 2016 at 12:13 PM, Jefferson Francisco < jeffersont...@gmail.com> wrote:
It is a clean install. I verified that it is v1.4.8
Today, I logged into Aggregate and updated my api certificate and keys.
Then I published the 2 submissions. I have probably spent a total of 30
minutes on Aggregate today. I also checked the first option on the bottom
of the Site admin / preferences page to slow down background jobs.Here are my quota stats for today:
Frontend Instance Hours 32%
Datastore Read Operations 11%
Datastore Write Operations 1%Does this look like normal quota usage?
On Sunday, May 8, 2016 at 12:27:13 PM UTC-6, Mitch Sundt wrote:
Is this a clean install or an upgrade from an earlier version?
If it is an upgrade, beginning with ODK Aggregate v1.4.8, we no longer
use backends.
So the error would suggest that there is a rogue backend on your system.I would recommend:
(0) Verify that you are running ODK Aggregate v1.4.8 by logging in with
Site Admin privileges and viewing the Site Admin / Preferences screen.(1) close all browser windows to your ODK Aggregate server.
(2) go to the AppEngine instances console:
https://console.cloud.google.com/appengine/instances
(3) Select your project / application ID in the top right of the screen.
(4) Then for the "default" and "background" instances, choose Shutdown
to stop them.(5) If you weren't running v1.4.8, upgrade to v1.4.8
The quota limits are reset daily, so this should clear fairly quickly.
On Sun, May 8, 2016 at 10:54 AM, Jefferson Francisco < jeffersont...@gmail.com> wrote:
1.4.8
On Sunday, May 8, 2016 at 11:48:13 AM UTC-6, Mitch Sundt wrote:
What version of ODK Aggregate are you using?
On Sat, May 7, 2016 at 4:41 PM, Jefferson Francisco < jeffersont...@gmail.com> wrote:
Here is what I found at
https://opendatakit.org/use/aggregate/deployment-planning/App Engine Usage Fees¶
https://opendatakit.org/use/aggregate/deployment-planning/#App_Engine_Usage_FeesOn App Engine, the primary consumption is Datastore Reads.
To minimize App Engine usage fees, users should restrict access to
the ODK Aggregate website and NOTkeep browser windows open on the
submissions tab. That tab is refreshed every 6 seconds if the user is
actively doing work, or slightly less frequently if they are not. At the
default display of 100 submissions (a minimum of 100 Reads), that can
quickly add up. Keep in mind that every select-one or select-multiple
question incurs an additional Read (one for each value stored). And images
incur a minimum of 10 Reads. Every 200 questions in your survey costs an
additional Read, and each repeat group also costs a Read for each filled-in
repeat in that group.On Saturday, May 7, 2016 at 5:28:49 PM UTC-6, Jefferson Francisco wrote:
I have set up Aggregate on Google App Engine. I added two
submissions and then pushed them to google fusion tables for testing
purposes. Now, when I try to visit Aggregate it says that I am over my
quota or having an error. I looked at my quotas and the Backend Instance
Hours is over the quota.Does anyone know what could be causing me to go over the quota?
Backend Instance Hours
9.02 of 9 Instance Hours
100%
Standard--
Post: opend...@googlegroups.com
Unsubscribe: opendatakit...@googlegroups.com
Options: http://groups.google.com/group/opendatakit?hl=en
You received this message because you are subscribed to the Google
Groups "ODK Community" group.
To unsubscribe from this group and stop receiving emails from it,
send an email to opendatakit...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.--
Mitch Sundt
Software Engineer
University of Washington
mitche...@gmail.com--
Post: opend...@googlegroups.com
Unsubscribe: opendatakit...@googlegroups.com
Options: http://groups.google.com/group/opendatakit?hl=en
You received this message because you are subscribed to the Google
Groups "ODK Community" group.
To unsubscribe from this group and stop receiving emails from it, send
an email to opendatakit...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.--
Mitch Sundt
Software Engineer
University of Washington
mitche...@gmail.com--
Post: opend...@googlegroups.com
Unsubscribe: opendatakit...@googlegroups.com
Options: http://groups.google.com/group/opendatakit?hl=en
You received this message because you are subscribed to the Google
Groups "ODK Community" group.
To unsubscribe from this group and stop receiving emails from it, send
an email to opendatakit...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.--
Mitch Sundt
Software Engineer
University of Washington
mitche...@gmail.com--
Post: opendatakit@googlegroups.com
Unsubscribe: opendatakit+unsubscribe@googlegroups.com
Options: http://groups.google.com/group/opendatakit?hl=en
You received this message because you are subscribed to the Google Groups
"ODK Community" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to opendatakit+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
--
Mitch Sundt
Software Engineer
University of Washington
mitchellsundt@gmail.com