Local aggregate server sizing

Hi list. I'm looking for guidelines for specifying a local server that is
expected to handle at least 200,000 records, perhaps more. Are there
documentations for server estimation?

TIA

··· -- -- Nik Need help customizing Survey Instruments/Tools/Forms for ODK Collect? Drop me an email.

Hi Nik,

If you can, run on GAE then you don't have to worry about scale -- it
just happens automagically.

If you want to run locally, I don't think the number of records matter
as much as what is in those records and how quickly they'll be hitting
the server. For example, if you have 200k records and there are 1Mb of
pictures, that's going to be 200 GB of storage. Or if you are going to
be handling 1000s of enumerators simultaneously, then you'll need a
fair bit of RAM and CPU.

The SurveyCTO team spins up servers on AWS regularly, so maybe Chris
can offer some rough numbers when he has spare cycles.

Yaw

··· -- Need ODK services? http://nafundi.com provides form design, server setup, professional support, and software development for ODK.

On Wed, May 21, 2014 at 5:07 PM, ゴー・ニコライ nikolai.go@gmail.com wrote:

Hi list. I'm looking for guidelines for specifying a local server that is
expected to handle at least 200,000 records, perhaps more. Are there
documentations for server estimation?

TIA

--

Nik
Need help customizing Survey Instruments/Tools/Forms for ODK Collect? Drop
me an email.

--

Post: opendatakit@googlegroups.com
Unsubscribe: opendatakit+unsubscribe@googlegroups.com
Options: http://groups.google.com/group/opendatakit?hl=en


You received this message because you are subscribed to the Google Groups
"ODK Community" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to opendatakit+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Unfortunately, we can't really give any good estimates here since (a) we've
tuned resource use in the SurveyCTO servers, and (b) we pool resources
across many virtual servers so that there's always a large amount of excess
capacity. I will say that the most intensive operations are actually adding
and deleting large forms, in terms of RAM, CPU power, and DB bandwidth. So
I'd put together a massive, several-thousand-field form with lots of media
fields and test your set-up by adding and deleting versions of that form.
The actual business of serving up forms, accepting submissions, and
delivering submissions is lightweight in comparison.

Best,

Chris

··· On Tue, May 27, 2014 at 10:32 AM, Yaw Anokwa wrote:

Hi Nik,

If you can, run on GAE then you don't have to worry about scale -- it
just happens automagically.

If you want to run locally, I don't think the number of records matter
as much as what is in those records and how quickly they'll be hitting
the server. For example, if you have 200k records and there are 1Mb of
pictures, that's going to be 200 GB of storage. Or if you are going to
be handling 1000s of enumerators simultaneously, then you'll need a
fair bit of RAM and CPU.

The SurveyCTO team spins up servers on AWS regularly, so maybe Chris
can offer some rough numbers when he has spare cycles.

Yaw

Need ODK services? http://nafundi.com provides form design, server
setup, professional support, and software development for ODK.

On Wed, May 21, 2014 at 5:07 PM, ゴー・ニコライ nikolai.go@gmail.com wrote:

Hi list. I'm looking for guidelines for specifying a local server that is
expected to handle at least 200,000 records, perhaps more. Are there
documentations for server estimation?

TIA

--

Nik
Need help customizing Survey Instruments/Tools/Forms for ODK Collect?
Drop
me an email.

--

Post: opendatakit@googlegroups.com
Unsubscribe: opendatakit+unsubscribe@googlegroups.com
Options: http://groups.google.com/group/opendatakit?hl=en


You received this message because you are subscribed to the Google Groups
"ODK Community" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to opendatakit+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--

Post: opendatakit@googlegroups.com
Unsubscribe: opendatakit+unsubscribe@googlegroups.com
Options: http://groups.google.com/group/opendatakit?hl=en


You received this message because you are subscribed to the Google Groups
"ODK Community" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to opendatakit+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Thanks Yaw and Christ for the feedback.

··· On Tuesday, May 27, 2014, Christopher Robert wrote:

Unfortunately, we can't really give any good estimates here since (a)
we've tuned resource use in the SurveyCTO servers, and (b) we pool
resources across many virtual servers so that there's always a large amount
of excess capacity. I will say that the most intensive operations are
actually adding and deleting large forms, in terms of RAM, CPU power, and
DB bandwidth. So I'd put together a massive, several-thousand-field form
with lots of media fields and test your set-up by adding and deleting
versions of that form. The actual business of serving up forms, accepting
submissions, and delivering submissions is lightweight in comparison.

Best,

Chris

On Tue, May 27, 2014 at 10:32 AM, Yaw Anokwa <yanokwa@nafundi.com <javascript:_e(%7B%7D,'cvml','yanokwa@nafundi.com');>> wrote:

Hi Nik,

If you can, run on GAE then you don't have to worry about scale -- it
just happens automagically.

If you want to run locally, I don't think the number of records matter
as much as what is in those records and how quickly they'll be hitting
the server. For example, if you have 200k records and there are 1Mb of
pictures, that's going to be 200 GB of storage. Or if you are going to
be handling 1000s of enumerators simultaneously, then you'll need a
fair bit of RAM and CPU.

The SurveyCTO team spins up servers on AWS regularly, so maybe Chris
can offer some rough numbers when he has spare cycles.

Yaw

Need ODK services? http://nafundi.com provides form design, server
setup, professional support, and software development for ODK.

On Wed, May 21, 2014 at 5:07 PM, ゴー・ニコライ <nikolai.go@gmail.com <javascript:_e(%7B%7D,'cvml','nikolai.go@gmail.com');>> wrote:

Hi list. I'm looking for guidelines for specifying a local server that
is
expected to handle at least 200,000 records, perhaps more. Are there
documentations for server estimation?

TIA

--

Nik
Need help customizing Survey Instruments/Tools/Forms for ODK Collect?
Drop
me an email.

--

Post: opendatakit@googlegroups.com
<javascript:_e(%7B%7D,'cvml','opendatakit@googlegroups.com');>
Unsubscribe: opendatakit+unsubscribe@googlegroups.com
<javascript:_e(%7B%7D,'cvml','opendatakit%2Bunsubscribe@googlegroups.com');>
Options: http://groups.google.com/group/opendatakit?hl=en


You received this message because you are subscribed to the Google
Groups
"ODK Community" group.
To unsubscribe from this group and stop receiving emails from it, send
an
email to opendatakit+unsubscribe@googlegroups.com
<javascript:_e(%7B%7D,'cvml','opendatakit%2Bunsubscribe@googlegroups.com');>
.
For more options, visit https://groups.google.com/d/optout.

--

Post: opendatakit@googlegroups.com
<javascript:_e(%7B%7D,'cvml','opendatakit@googlegroups.com');>
Unsubscribe: opendatakit+unsubscribe@googlegroups.com
<javascript:_e(%7B%7D,'cvml','opendatakit%2Bunsubscribe@googlegroups.com');>
Options: http://groups.google.com/group/opendatakit?hl=en


You received this message because you are subscribed to the Google Groups
"ODK Community" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to opendatakit+unsubscribe@googlegroups.com
<javascript:_e(%7B%7D,'cvml','opendatakit%2Bunsubscribe@googlegroups.com');>
.
For more options, visit https://groups.google.com/d/optout.

--

Post: opendatakit@googlegroups.com
<javascript:_e(%7B%7D,'cvml','opendatakit@googlegroups.com');>
Unsubscribe: opendatakit+unsubscribe@googlegroups.com
<javascript:_e(%7B%7D,'cvml','opendatakit%2Bunsubscribe@googlegroups.com');>
Options: http://groups.google.com/group/opendatakit?hl=en


You received this message because you are subscribed to the Google Groups
"ODK Community" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to opendatakit+unsubscribe@googlegroups.com
<javascript:_e(%7B%7D,'cvml','opendatakit%2Bunsubscribe@googlegroups.com');>
.
For more options, visit https://groups.google.com/d/optout.

--

Nik
Need help customizing Survey Instruments/Tools/Forms for ODK Collect? Drop
me an email.