Hello All:
Please help with the following information:
- How many forms can be uploaded to an aggregate server (if using goggle paid app Engine)?
- what is the maximum limit of forms that can be uploaded per minutes /day?
I have uploaded about 600 forms in a server in 3 slots using ODK briefcase. After the third upload of about 200 forms (which went really slow), my server (of Google paid account) wasn't responding. Neither I can see uploaded forms from the server nor can get those on phone using odk collect. Server says the per minute/day limit has exceeded.
Google attempts to protect your AppEngine servers from bursts of activity
that might otherwise deplete your daily billing quota. If Google detects a
burst of activity that is in some proportion to the daily quota on the
service, it will start to deny or throttle back that burst.
The intent is to protect your investment from malicious or misbehaving apps
that interact with it.
See https://cloud.google.com/appengine/docs/quotas?hl=en
···
----------------------
Are these 600 form definitions or 600 submissions?
Form definitions consume server memory because they are always held in
memory.
Submissions are only read from the database when they are referenced.
Aggregate can easily manage 1000's of submissions.
But as the number of form definitions increases, they deplete more and more
server memory, causing the server to run slower and slower. Eventually, the
memory size of your AppEngine instance must be increased to restore the
server's responsiveness. When this is necessary will depend upon the size
of the forms. I would expect this to become necessary at around 60 to 100
form definitions of moderate size.
Increasing the memory size currently requires many manual steps and a
redeployment to AppEngine.
On Mon, Oct 19, 2015 at 9:09 AM, Ehsan Haque ehuqtamal@gmail.com wrote:
Hello All:
Please help with the following information:
- How many forms can be uploaded to an aggregate server (if using goggle
paid app Engine)?
- what is the maximum limit of forms that can be uploaded per minutes
/day?
I have uploaded about 600 forms in a server in 3 slots using ODK
briefcase. After the third upload of about 200 forms (which went really
slow), my server (of Google paid account) wasn't responding. Neither I can
see uploaded forms from the server nor can get those on phone using odk
collect. Server says the per minute/day limit has exceeded.
--
Post: opendatakit@googlegroups.com
Unsubscribe: opendatakit+unsubscribe@googlegroups.com
Options: http://groups.google.com/group/opendatakit?hl=en
You received this message because you are subscribed to the Google Groups
"ODK Community" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to opendatakit+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
--
Mitch Sundt
Software Engineer
University of Washington
mitchellsundt@gmail.com
Thanks for your reply.
I have uploaded small to medium sized 600 form definitions (some of the
last attempts were failed/partially successful). Now I cant access those
from the server. Will it be available for accessing after one day? How can
I efficiently manage these 600 forms into a server?
···
On Monday, October 19, 2015 at 11:41:03 PM UTC+6, Mitch Sundt wrote:
>
> Google attempts to protect your AppEngine servers from bursts of activity
> that might otherwise deplete your daily billing quota. If Google detects a
> burst of activity that is in some proportion to the daily quota on the
> service, it will start to deny or throttle back that burst.
>
> The intent is to protect your investment from malicious or misbehaving
> apps that interact with it.
>
> See https://cloud.google.com/appengine/docs/quotas?hl=en
>
> ----------------------
> Are these 600 form definitions or 600 submissions?
>
> Form definitions consume server memory because they are always held in
> memory.
> Submissions are only read from the database when they are referenced.
>
> Aggregate can easily manage 1000's of submissions.
>
> But as the number of form definitions increases, they deplete more and
> more server memory, causing the server to run slower and slower.
> Eventually, the memory size of your AppEngine instance must be increased to
> restore the server's responsiveness. When this is necessary will depend
> upon the size of the forms. I would expect this to become necessary at
> around 60 to 100 form definitions of moderate size.
>
> Increasing the memory size currently requires many manual steps and a
> redeployment to AppEngine.
>
>
> On Mon, Oct 19, 2015 at 9:09 AM, Ehsan Haque <ehuq...@gmail.com > wrote:
>
>> Hello All:
>>
>> Please help with the following information:
>>
>> 1) How many forms can be uploaded to an aggregate server (if using goggle
>> paid app Engine)?
>> 2) what is the maximum limit of forms that can be uploaded per minutes
>> /day?
>>
>> I have uploaded about 600 forms in a server in 3 slots using ODK
>> briefcase. After the third upload of about 200 forms (which went really
>> slow), my server (of Google paid account) wasn't responding. Neither I can
>> see uploaded forms from the server nor can get those on phone using odk
>> collect. Server says the per minute/day limit has exceeded.
>>
>> --
>> --
>> Post: opend...@googlegroups.com
>> Unsubscribe: opendatakit...@googlegroups.com
>> Options: http://groups.google.com/group/opendatakit?hl=en
>>
>> ---
>> You received this message because you are subscribed to the Google Groups
>> "ODK Community" group.
>> To unsubscribe from this group and stop receiving emails from it, send an
>> email to opendatakit...@googlegroups.com .
>> For more options, visit https://groups.google.com/d/optout.
>>
>
>
>
> --
> Mitch Sundt
> Software Engineer
> University of Washington
> mitche...@gmail.com
>
With that many form definitions on one server, you CANNOT run for free.
You will need to enable billing (your free quota will be consumed in 15
minutes or less).
And you will likely need to increase the size of your servers.
Why do you need 600 form definitions?
The typical use case is to construct one form definition, use that across
100's of devices, and send all the collected data back into ODK Aggregate
into one submissions table.
If you wanted to pre-fill fields in a form for a specific data collector,
people generally use the data preload feature
https://opendatakit.org/help/form-design/data-preloading/ or an external
itemset from which the data collector would pick their identity during the
filling-out of the survey.
Having a need for 600 form definitions suggests you are doing something
wrong in the design of these forms.
···
On Mon, Oct 19, 2015 at 8:50 PM, Ehsan Haque wrote:
Thanks for your reply.
I have uploaded small to medium sized 600 form definitions (some of the
last attempts were failed/partially successful). Now I cant access those
from the server. Will it be available for accessing after one day? How can
I efficiently manage these 600 forms into a server?
On Monday, October 19, 2015 at 11:41:03 PM UTC+6, Mitch Sundt wrote:
Google attempts to protect your AppEngine servers from bursts of activity
that might otherwise deplete your daily billing quota. If Google detects a
burst of activity that is in some proportion to the daily quota on the
service, it will start to deny or throttle back that burst.
The intent is to protect your investment from malicious or misbehaving
apps that interact with it.
See https://cloud.google.com/appengine/docs/quotas?hl=en
Are these 600 form definitions or 600 submissions?
Form definitions consume server memory because they are always held in
memory.
Submissions are only read from the database when they are referenced.
Aggregate can easily manage 1000's of submissions.
But as the number of form definitions increases, they deplete more and
more server memory, causing the server to run slower and slower.
Eventually, the memory size of your AppEngine instance must be increased to
restore the server's responsiveness. When this is necessary will depend
upon the size of the forms. I would expect this to become necessary at
around 60 to 100 form definitions of moderate size.
Increasing the memory size currently requires many manual steps and a
redeployment to AppEngine.
On Mon, Oct 19, 2015 at 9:09 AM, Ehsan Haque ehuq...@gmail.com wrote:
Hello All:
Please help with the following information:
- How many forms can be uploaded to an aggregate server (if using
goggle paid app Engine)?
- what is the maximum limit of forms that can be uploaded per minutes
/day?
I have uploaded about 600 forms in a server in 3 slots using ODK
briefcase. After the third upload of about 200 forms (which went really
slow), my server (of Google paid account) wasn't responding. Neither I can
see uploaded forms from the server nor can get those on phone using odk
collect. Server says the per minute/day limit has exceeded.
--
Post: opend...@googlegroups.com
Unsubscribe: opendatakit...@googlegroups.com
Options: http://groups.google.com/group/opendatakit?hl=en
You received this message because you are subscribed to the Google
Groups "ODK Community" group.
To unsubscribe from this group and stop receiving emails from it, send
an email to opendatakit...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
--
Mitch Sundt
Software Engineer
University of Washington
mitche...@gmail.com
--
Post: opendatakit@googlegroups.com
Unsubscribe: opendatakit+unsubscribe@googlegroups.com
Options: http://groups.google.com/group/opendatakit?hl=en
You received this message because you are subscribed to the Google Groups
"ODK Community" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to opendatakit+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
--
Mitch Sundt
Software Engineer
University of Washington
mitchellsundt@gmail.com
I'm using a paid google account. There might be some settings which I have to change to increase my quota, not sure though.
The reason for planning to use so many form definition is, we would like to appear same form in different points of time at the user's launcher. So most of the same forms contains a different prefill to pop up at different time. Any suggestion?
Seems like there are two requirements:
(1) ability to begin a new form at time X
(2) figuring out, for a given submission, what time X was.
For (1), configure the device with an alarm to remind the data collector to
begin a new survey; train the data collector to "Fill Blank Form" and
select the form.
For (2), you can use the start time preloaded field
https://opendatakit.org/help/form-design/examples/#preload_params to record
the time the form was created (or, on http://xlsform.org/ , search down for
'metadata').
During data analysis, you would use this start time field that is
automatically recorded in the submission to place each submission into your
set of time bins. e.g., into bin X if the submission start time fell
between start time A and start time B.
If you were exporting data to CSV, this could be a calculated column in
your Excel spreadsheet that transformed the start time field into an
integer value (0-599).
This would allow you to use a single form during data collection.
···
On Tue, Oct 20, 2015 at 10:07 AM, Ehsan Haque wrote:
I'm using a paid google account. There might be some settings which I have
to change to increase my quota, not sure though.
The reason for planning to use so many form definition is, we would like
to appear same form in different points of time at the user's launcher. So
most of the same forms contains a different prefill to pop up at different
time. Any suggestion?
--
Post: opendatakit@googlegroups.com
Unsubscribe: opendatakit+unsubscribe@googlegroups.com
Options: http://groups.google.com/group/opendatakit?hl=en
You received this message because you are subscribed to the Google Groups
"ODK Community" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to opendatakit+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
--
Mitch Sundt
Software Engineer
University of Washington
mitchellsundt@gmail.com