We're about to go LIVE on a survey with 20 Android devices. Survey is
mostly multi selects, yes/no and some data entry (Name, Place, Comments)
and one pic (I've changed the default settings in Android camera to the
smallest resolution) of the area. Can someone give me an idea what I can
expect for a 3000 respondent survey over a 20 day period (target is 4
submissions per hour; seems like a typical xml instance file size is 1007
bytes from file manager Details & pic is 60K).
Do you think we can get by with the 1GB data storage allocation from
AppEngine? Will this FREE application handle the form submissions WHEN
required? I'm thinking of buying some credits, what would you suggest?
Also, I saw settings for F1, F2, F4 in the Application tab for utilising
more server memory/processor speed. Is the basic F1 enough?
Can some other ODK users give me an idea (or share their thoughts) of some
surveys they've succesfully completed WITHIN the quota of the 1GB limit? We
just don't want egg on our faces with this project. 
Cheers!
~DataMax
We have some information about people's experiences on our website:
http://opendatakit.org/use/aggregate/deployment-planning/
In our experience if you enable billing most people do not exceed the
the minimum charge of $2.10/week unless there is lots of publishing to
many external services. Therefore if your group can afford it spending
the $2.10/week for piece of mind that you won't run out of resources
it maybe worth it so you don't have to worry.
The instance we run for people to play with (opendatakit.appspot.com)
uses the F1 instance just fine and has a maximum daily budget limit of
$1.50. If we didn't have multiple people setting up publishing
services that they don't care about and let keep running (we go purge
them every once in awhile) we would only spend the $2.10/week.
However, because there are lots of background forwarding that users
experimenting have setup and then leave we exceed that.
Also keep in mind the more you use the UI the more resources you will
use on your instance.
Waylon
···
On Fri, May 18, 2012 at 9:33 PM, DataMax wrote:
> We're about to go LIVE on a survey with 20 Android devices. Survey is mostly
> multi selects, yes/no and some data entry (Name, Place, Comments) and one
> pic (I've changed the default settings in Android camera to the smallest
> resolution) of the area. Can someone give me an idea what I can expect for a
> 3000 respondent survey over a 20 day period (target is 4 submissions per
> hour; seems like a typical xml instance file size is 1007 bytes from file
> manager Details & pic is 60K).
>
> Do you think we can get by with the 1GB data storage allocation from
> AppEngine? Will this FREE application handle the form submissions WHEN
> required? I'm thinking of buying some credits, what would you suggest? Also,
> I saw settings for F1, F2, F4 in the Application tab for utilising more
> server memory/processor speed. Is the basic F1 enough?
> Can some other ODK users give me an idea (or share their thoughts) of some
> surveys they've succesfully completed WITHIN the quota of the 1GB limit? We
> just don't want egg on our faces with this project. :-)
>
> Cheers!
>
> ~DataMax
>
> --
> Post: opendatakit@googlegroups.com
> Unsubscribe: opendatakit+unsubscribe@googlegroups.com
> Options: http://groups.google.com/group/opendatakit?hl=en
Hi Waylon!
Thanks for the comment! I will activate billing ASAP for the instance.
The survey is 4 times per year, so it's only during that 2 week period.
What kind of traffic (page requests, storage, etc.) is the OpenDataKit
application getting? Any ideas how many mobile devices connect at one time
to submit forms using an F1 setup?
~DataMax
···
--------------------------
On Saturday, May 19, 2012 10:23:11 PM UTC+3, waylon wrote:
We have some information about people's experiences on our website:
http://opendatakit.org/use/aggregate/deployment-planning/
In our experience if you enable billing most people do not exceed the
the minimum charge of $2.10/week unless there is lots of publishing to
many external services. Therefore if your group can afford it spending
the $2.10/week for piece of mind that you won't run out of resources
it maybe worth it so you don't have to worry.
The instance we run for people to play with (opendatakit.appspot.com)
uses the F1 instance just fine and has a maximum daily budget limit of
$1.50. If we didn't have multiple people setting up publishing
services that they don't care about and let keep running (we go purge
them every once in awhile) we would only spend the $2.10/week.
However, because there are lots of background forwarding that users
experimenting have setup and then leave we exceed that.
Also keep in mind the more you use the UI the more resources you will
use on your instance.
Waylon
On Fri, May 18, 2012 at 9:33 PM, DataMax maxtheitpro@gmail.com wrote:
We're about to go LIVE on a survey with 20 Android devices. Survey is
mostly
multi selects, yes/no and some data entry (Name, Place, Comments) and
one
pic (I've changed the default settings in Android camera to the smallest
resolution) of the area. Can someone give me an idea what I can expect
for a
3000 respondent survey over a 20 day period (target is 4 submissions per
hour; seems like a typical xml instance file size is 1007 bytes from
file
manager Details & pic is 60K).
Do you think we can get by with the 1GB data storage allocation from
AppEngine? Will this FREE application handle the form submissions WHEN
required? I'm thinking of buying some credits, what would you suggest?
Also,
I saw settings for F1, F2, F4 in the Application tab for utilising more
server memory/processor speed. Is the basic F1 enough?
Can some other ODK users give me an idea (or share their thoughts) of
some
surveys they've succesfully completed WITHIN the quota of the 1GB limit?
We
just don't want egg on our faces with this project.
Cheers!
~DataMax
--
Post: opendatakit@googlegroups.com
Unsubscribe: opendatakit+unsubscribe@googlegroups.com
Options: http://groups.google.com/group/opendatakit?hl=en
If you are using Google App Engine your question should not matter as
it doesn't really make sense. AppEngine should detect if one instance
of the foreground server is being overworked and will start a second
instance of an F1 server, then a third instance to handle the traffic
if needed. Hence why they say "Scale your app automatically without
worrying about managing machines." They then bill you for the cost of
the CPU time so if you get all your requests in an hour they will
"start up" enough instances to handle those requests and then will
terminate those server instances when they are not being used.
Notice how that AppEngine bills you based on the CPU time used not
number of machines used. NOTE: this means that if there has been no
request in a long enough period of time there will be no servers
running and on the first request AppEngine will "start up" a new
server instance and this can cause a substantial delay and lead to a
timeout. That is why often the first request to Aggregate is really
slow and then afterwards it responds much faster. The tools have
automatic retries built in to handle this situation but sometimes
manual retries are good.
AppEngine also allows you to keep the servers running and guarantees
they won't terminate them but they charge for that and it's expensive.
You should read about AppEngine on their website because I cannot
cover every detail in email.
Hope this briefly answer your question that you don't need to worry
about the front end server instance size. Adjusting the backend size
can help when generating export files because you can run out of ram
for really big files.
Waylon
···
On Sat, May 19, 2012 at 4:08 PM, DataMax wrote:
> Hi Waylon!
> Thanks for the comment! I will activate billing ASAP for the instance. :-)
> The survey is 4 times per year, so it's only during that 2 week period. What
> kind of traffic (page requests, storage, etc.) is the OpenDataKit
> application getting? Any ideas how many mobile devices connect at one time
> to submit forms using an F1 setup?
>
> ~DataMax
> --------------------------
>
>
>
> On Saturday, May 19, 2012 10:23:11 PM UTC+3, waylon wrote:
>>
>> We have some information about people's experiences on our website:
>> http://opendatakit.org/use/aggregate/deployment-planning/
>>
>> In our experience if you enable billing most people do not exceed the
>> the minimum charge of $2.10/week unless there is lots of publishing to
>> many external services. Therefore if your group can afford it spending
>> the $2.10/week for piece of mind that you won't run out of resources
>> it maybe worth it so you don't have to worry.
>>
>> The instance we run for people to play with (opendatakit.appspot.com)
>> uses the F1 instance just fine and has a maximum daily budget limit of
>> $1.50. If we didn't have multiple people setting up publishing
>> services that they don't care about and let keep running (we go purge
>> them every once in awhile) we would only spend the $2.10/week.
>> However, because there are lots of background forwarding that users
>> experimenting have setup and then leave we exceed that.
>>
>> Also keep in mind the more you use the UI the more resources you will
>> use on your instance.
>>
>> Waylon
>>
>> On Fri, May 18, 2012 at 9:33 PM, DataMax wrote:
>> > We're about to go LIVE on a survey with 20 Android devices. Survey is
>> > mostly
>> > multi selects, yes/no and some data entry (Name, Place, Comments) and
>> > one
>> > pic (I've changed the default settings in Android camera to the smallest
>> > resolution) of the area. Can someone give me an idea what I can expect
>> > for a
>> > 3000 respondent survey over a 20 day period (target is 4 submissions per
>> > hour; seems like a typical xml instance file size is 1007 bytes from
>> > file
>> > manager Details & pic is 60K).
>> >
>> > Do you think we can get by with the 1GB data storage allocation from
>> > AppEngine? Will this FREE application handle the form submissions WHEN
>> > required? I'm thinking of buying some credits, what would you suggest?
>> > Also,
>> > I saw settings for F1, F2, F4 in the Application tab for utilising more
>> > server memory/processor speed. Is the basic F1 enough?
>> > Can some other ODK users give me an idea (or share their thoughts) of
>> > some
>> > surveys they've succesfully completed WITHIN the quota of the 1GB limit?
>> > We
>> > just don't want egg on our faces with this project. :-)
>> >
>> > Cheers!
>> >
>> > ~DataMax
>> >
>> > --
>> > Post: opendatakit@googlegroups.com
>> > Unsubscribe: opendatakit+unsubscribe@googlegroups.com
>> > Options: http://groups.google.com/group/opendatakit?hl=en
>
> --
> Post: opendatakit@googlegroups.com
> Unsubscribe: opendatakit+unsubscribe@googlegroups.com
> Options: http://groups.google.com/group/opendatakit?hl=en
Ah, that explanation helped a LOT!
Thanks a million! I'm not as worried any more.
~DataMax
···
-----------------------
On Sunday, May 20, 2012 7:45:39 AM UTC+3, waylon wrote:
If you are using Google App Engine your question should not matter as
it doesn't really make sense. AppEngine should detect if one instance
of the foreground server is being overworked and will start a second
instance of an F1 server, then a third instance to handle the traffic
if needed. Hence why they say "Scale your app automatically without
worrying about managing machines." They then bill you for the cost of
the CPU time so if you get all your requests in an hour they will
"start up" enough instances to handle those requests and then will
terminate those server instances when they are not being used.
Notice how that AppEngine bills you based on the CPU time used not
number of machines used. NOTE: this means that if there has been no
request in a long enough period of time there will be no servers
running and on the first request AppEngine will "start up" a new
server instance and this can cause a substantial delay and lead to a
timeout. That is why often the first request to Aggregate is really
slow and then afterwards it responds much faster. The tools have
automatic retries built in to handle this situation but sometimes
manual retries are good.
AppEngine also allows you to keep the servers running and guarantees
they won't terminate them but they charge for that and it's expensive.
You should read about AppEngine on their website because I cannot
cover every detail in email.
https://developers.google.com/appengine/
Hope this briefly answer your question that you don't need to worry
about the front end server instance size. Adjusting the backend size
can help when generating export files because you can run out of ram
for really big files.
Waylon
On Sat, May 19, 2012 at 4:08 PM, DataMax maxtheitpro@gmail.com wrote:
Hi Waylon!
Thanks for the comment! I will activate billing ASAP for the instance.
The survey is 4 times per year, so it's only during that 2 week period.
What
kind of traffic (page requests, storage, etc.) is the OpenDataKit
application getting? Any ideas how many mobile devices connect at one
time
to submit forms using an F1 setup?
~DataMax
On Saturday, May 19, 2012 10:23:11 PM UTC+3, waylon wrote:
We have some information about people's experiences on our website:
http://opendatakit.org/use/aggregate/deployment-planning/
In our experience if you enable billing most people do not exceed the
the minimum charge of $2.10/week unless there is lots of publishing to
many external services. Therefore if your group can afford it spending
the $2.10/week for piece of mind that you won't run out of resources
it maybe worth it so you don't have to worry.
The instance we run for people to play with (opendatakit.appspot.com)
uses the F1 instance just fine and has a maximum daily budget limit of
$1.50. If we didn't have multiple people setting up publishing
services that they don't care about and let keep running (we go purge
them every once in awhile) we would only spend the $2.10/week.
However, because there are lots of background forwarding that users
experimenting have setup and then leave we exceed that.
Also keep in mind the more you use the UI the more resources you will
use on your instance.
Waylon
On Fri, May 18, 2012 at 9:33 PM, DataMax maxtheitpro@gmail.com wrote:
We're about to go LIVE on a survey with 20 Android devices. Survey is
mostly
multi selects, yes/no and some data entry (Name, Place, Comments) and
one
pic (I've changed the default settings in Android camera to the
smallest
resolution) of the area. Can someone give me an idea what I can
expect
for a
3000 respondent survey over a 20 day period (target is 4 submissions
per
hour; seems like a typical xml instance file size is 1007 bytes from
file
manager Details & pic is 60K).
Do you think we can get by with the 1GB data storage allocation from
AppEngine? Will this FREE application handle the form submissions
WHEN
required? I'm thinking of buying some credits, what would you
suggest?
Also,
I saw settings for F1, F2, F4 in the Application tab for utilising
more
server memory/processor speed. Is the basic F1 enough?
Can some other ODK users give me an idea (or share their thoughts) of
some
surveys they've succesfully completed WITHIN the quota of the 1GB
limit?
We
just don't want egg on our faces with this project.
Cheers!
~DataMax
--
Post: opendatakit@googlegroups.com
Unsubscribe: opendatakit+unsubscribe@googlegroups.com
Options: http://groups.google.com/group/opendatakit?hl=en
--
Post: opendatakit@googlegroups.com
Unsubscribe: opendatakit+unsubscribe@googlegroups.com
Options: http://groups.google.com/group/opendatakit?hl=en
As a sizing data point, I have an ODK Aggregate instance with:
- Geo Tagger v2 form with a dataset of 600 filled-in surveys with images
ranging from 750KB to 1600KB in size.
- various Miramare and other blank forms with a combined total of 533 media
files referenced by those blank forms.
With all of that on the instance, it uses about 0.75 GB of space on
AppEngine.
Note that 600 * avg( 750KB,1600KB ) = 705MB.
And is is quite reasonable to expect that the 533 media files add another
45MB, accounting for the 750MB (0.75GB) reported by Google.
In general, the size of the picture will dominate the size of any text
survey, so if you know your images are around 250KB, you can estimate your
storage requirements, for 3000 surveys, by 3000 * 250KB = 750MB.
Mitch
And for size comparisons, I have an ODK Aggregate instance with a Geo
Tagger v2 form with
···
On Sun, May 20, 2012 at 1:53 AM, DataMax wrote:
Ah, that explanation helped a LOT!
Thanks a million! I'm not as worried any more.
~DataMax
On Sunday, May 20, 2012 7:45:39 AM UTC+3, waylon wrote:
If you are using Google App Engine your question should not matter as
it doesn't really make sense. AppEngine should detect if one instance
of the foreground server is being overworked and will start a second
instance of an F1 server, then a third instance to handle the traffic
if needed. Hence why they say "Scale your app automatically without
worrying about managing machines." They then bill you for the cost of
the CPU time so if you get all your requests in an hour they will
"start up" enough instances to handle those requests and then will
terminate those server instances when they are not being used.
Notice how that AppEngine bills you based on the CPU time used not
number of machines used. NOTE: this means that if there has been no
request in a long enough period of time there will be no servers
running and on the first request AppEngine will "start up" a new
server instance and this can cause a substantial delay and lead to a
timeout. That is why often the first request to Aggregate is really
slow and then afterwards it responds much faster. The tools have
automatic retries built in to handle this situation but sometimes
manual retries are good.
AppEngine also allows you to keep the servers running and guarantees
they won't terminate them but they charge for that and it's expensive.
You should read about AppEngine on their website because I cannot
cover every detail in email.
https://developers.google.com/**appengine/https://developers.google.com/appengine/
Hope this briefly answer your question that you don't need to worry
about the front end server instance size. Adjusting the backend size
can help when generating export files because you can run out of ram
for really big files.
Waylon
On Sat, May 19, 2012 at 4:08 PM, DataMax maxtheitpro@gmail.com wrote:
Hi Waylon!
Thanks for the comment! I will activate billing ASAP for the instance.
The survey is 4 times per year, so it's only during that 2 week period.
What
kind of traffic (page requests, storage, etc.) is the OpenDataKit
application getting? Any ideas how many mobile devices connect at one
time
to submit forms using an F1 setup?
~DataMax
On Saturday, May 19, 2012 10:23:11 PM UTC+3, waylon wrote:
We have some information about people's experiences on our website:
http://opendatakit.org/use/**aggregate/deployment-planning/http://opendatakit.org/use/aggregate/deployment-planning/
In our experience if you enable billing most people do not exceed the
the minimum charge of $2.10/week unless there is lots of publishing to
many external services. Therefore if your group can afford it spending
the $2.10/week for piece of mind that you won't run out of resources
it maybe worth it so you don't have to worry.
The instance we run for people to play with (opendatakit.appspot.com)
uses the F1 instance just fine and has a maximum daily budget limit of
$1.50. If we didn't have multiple people setting up publishing
services that they don't care about and let keep running (we go purge
them every once in awhile) we would only spend the $2.10/week.
However, because there are lots of background forwarding that users
experimenting have setup and then leave we exceed that.
Also keep in mind the more you use the UI the more resources you will
use on your instance.
Waylon
On Fri, May 18, 2012 at 9:33 PM, DataMax maxtheitpro@gmail.com wrote:
We're about to go LIVE on a survey with 20 Android devices. Survey
is
mostly
multi selects, yes/no and some data entry (Name, Place, Comments)
and
one
pic (I've changed the default settings in Android camera to the
smallest
resolution) of the area. Can someone give me an idea what I can
expect
for a
3000 respondent survey over a 20 day period (target is 4 submissions
per
hour; seems like a typical xml instance file size is 1007 bytes from
file
manager Details & pic is 60K).
Do you think we can get by with the 1GB data storage allocation from
AppEngine? Will this FREE application handle the form submissions
WHEN
required? I'm thinking of buying some credits, what would you
suggest?
Also,
I saw settings for F1, F2, F4 in the Application tab for utilising
more
server memory/processor speed. Is the basic F1 enough?
Can some other ODK users give me an idea (or share their thoughts)
of
some
surveys they've succesfully completed WITHIN the quota of the 1GB
limit?
We
just don't want egg on our faces with this project.
Cheers!
~DataMax
--
Post: opendatakit@googlegroups.com
Unsubscribe: opendatakit+unsubscribe@**googlegroups.comopendatakit%2Bunsubscribe@googlegroups.com
Options: http://groups.google.com/**group/opendatakit?hl=enhttp://groups.google.com/group/opendatakit?hl=en
--
Post: opendatakit@googlegroups.com
Unsubscribe: opendatakit+unsubscribe@**googlegroups.comopendatakit%2Bunsubscribe@googlegroups.com
Options: http://groups.google.com/**group/opendatakit?hl=enhttp://groups.google.com/group/opendatakit?hl=en
--
Post: opendatakit@googlegroups.com
Unsubscribe: opendatakit+unsubscribe@googlegroups.com
Options: http://groups.google.com/group/opendatakit?hl=en
--
Mitch Sundt
Software Engineer
University of Washington
mitchellsundt@gmail.com