Better understanding the billing system for future deployment

Ok, thank you all for the additional information.

So it looks like we need to look at data usage using at least three
different levels: data storage space in total, bandwidth per day, number of
writes per day, number of read per day.

How can we calculate the number of writes and read per day? If I have a
filled form with 50 answers that is uploaded to the server, that will be 50
writes? If I visualize results on Aggregate, how do they calculate the
number of reads?

Maxim

··· On Wednesday, December 19, 2012 2:16:20 PM UTC-5, Maxim wrote: > > Hi, > > I am currently trying to better understand the billing system for a future > deployment using ODK on the Google AppEngine. The system would be used as > part of an environmental monitoring program for construction sites. > > Background information: > - 2 surveyors using Android phones > - 100 sites that would be visited over the course of 2 months > - 5 forms with an average of, let's say 20 questions, so a total of 100 > questions to be filled by site > - each entry would also have a low-resolution picture (about 200-300 kb in > size) > > I read the quotas and billing sections from the Help site but have some > trouble relating the information to what we are going to do. > > How can I use the background information to evaluate my future data usage? > I need to calculate the number of forms submitted per day, estimate the > total size of bandwidth used per day? I'm trying to see if I would be OK > with the minimum of 2.10$/week. > > Thanks, > > Maxim > >

The number of writes will generally not be a problem; you have 50,000 / day
for free.

The calculation for an individual submission is as follows:

  • one write for the submission,
  • one write for each filled-in repeat for every repeat group in that
    submission (if you have a household survey with 4 people (1st repeat
    group), and each person has 3 personal items (nested repeat group under
    'person'), then you would have 4 + 4x3 filled-in repeats -- for a total of
    16 writes),
  • one write for each selected item in any multiple-choice question,
  • 3 writes for each media (audio, video, image) attachment
  • 1 additional write for every 1,000,000 bytes of media attachment file
    size above the first 1,000,000 bytes.

For a very complex form with media, it is likely that you would have fewer
than 100 writes per submission. That would give you a daily budget of 500
submissions.

Not surprisingly, the number of required reads to access and display a
submission is the same as the number of writes.

But there are additional reads:

  • when you view the submissions, you are presented with 100 submissions
    per page
  • the submissions pages are refreshed every 70 seconds (until you do
    nothing for 3 minutes, after which it will sleep); the other pages refresh
    more frequently.
  • the form definition must be read and cached (that cache expires every
    3 seconds and the form definitions are re-read when next used). This is
    quite read-intensive, with one record per question and per group in the
    form.
  • the security information is re-read every 5 minutes to pick up the
    latest security settings changes
  • publishing requires reading each submission, and incurs 1 extra write
    per submission transmitted.
  • export to file requires reading each submission and incurs a handful
    of writes to save the formatted file.
  • a watchdog process kicks off every few minutes to detect stalled
    publishing or export jobs and restart them.

If you want to skim under the free limit, and find yourself hitting it, you
should minimize use of the website, and periodically do a pull of the data
using ODK Briefcase.

Mitch

··· On Fri, Dec 21, 2012 at 8:32 AM, Maxim wrote:

Ok, thank you all for the additional information.

So it looks like we need to look at data usage using at least three
different levels: data storage space in total, bandwidth per day, number of
writes per day, number of read per day.

How can we calculate the number of writes and read per day? If I have a
filled form with 50 answers that is uploaded to the server, that will be 50
writes? If I visualize results on Aggregate, how do they calculate the
number of reads?

Maxim

On Wednesday, December 19, 2012 2:16:20 PM UTC-5, Maxim wrote:

Hi,

I am currently trying to better understand the billing system for a
future deployment using ODK on the Google AppEngine. The system would be
used as part of an environmental monitoring program for construction sites.

Background information:

  • 2 surveyors using Android phones
  • 100 sites that would be visited over the course of 2 months
  • 5 forms with an average of, let's say 20 questions, so a total of 100
    questions to be filled by site
  • each entry would also have a low-resolution picture (about 200-300 kb
    in size)

I read the quotas and billing sections from the Help site but have some
trouble relating the information to what we are going to do.

How can I use the background information to evaluate my future data
usage? I need to calculate the number of forms submitted per day, estimate
the total size of bandwidth used per day? I'm trying to see if I would be
OK with the minimum of 2.10$/week.

Thanks,

Maxim

--
Post: opendatakit@googlegroups.com
Unsubscribe: opendatakit+unsubscribe@googlegroups.com
Options: http://groups.google.com/group/opendatakit?hl=en

--
Mitch Sundt
Software Engineer
University of Washington
mitchellsundt@gmail.com

Thanks a lot for the details Mitch. From your answers and from some testing
I did I would probably be fine under the free limit.

Maxim

··· On Friday, December 21, 2012 1:00:22 PM UTC-5, Mitch wrote: > > The number of writes will generally not be a problem; you have 50,000 / > day for free. > > The calculation for an individual submission is as follows: > > - one write for the submission, > - one write for each filled-in repeat for every repeat group in that > submission (if you have a household survey with 4 people (1st repeat > group), and each person has 3 personal items (nested repeat group under > 'person'), then you would have 4 + 4x3 filled-in repeats -- for a total of > 16 writes), > - one write for each selected item in any multiple-choice question, > - 3 writes for each media (audio, video, image) attachment > - 1 additional write for every 1,000,000 bytes of media attachment > file size above the first 1,000,000 bytes. > > For a very complex form with media, it is likely that you would have fewer > than 100 writes per submission. That would give you a daily budget of 500 > submissions. > > Not surprisingly, the number of required reads to access and display a > submission is the same as the number of writes. > > But there are additional reads: > > - when you view the submissions, you are presented with 100 > submissions per page > - the submissions pages are refreshed every 70 seconds (until you do > nothing for 3 minutes, after which it will sleep); the other pages refresh > more frequently. > - the form definition must be read and cached (that cache expires > every 3 seconds and the form definitions are re-read when next used). This > is quite read-intensive, with one record per question and per group in the > form. > - the security information is re-read every 5 minutes to pick up the > latest security settings changes > - publishing requires reading each submission, and incurs 1 extra > write per submission transmitted. > - export to file requires reading each submission and incurs a handful > of writes to save the formatted file. > - a watchdog process kicks off every few minutes to detect stalled > publishing or export jobs and restart them. > > If you want to skim under the free limit, and find yourself hitting it, > you should minimize use of the website, and periodically do a pull of the > data using ODK Briefcase. > > Mitch > > > > > On Fri, Dec 21, 2012 at 8:32 AM, Maxim <maxfr...@gmail.com >wrote: > >> Ok, thank you all for the additional information. >> >> So it looks like we need to look at data usage using at least three >> different levels: data storage space in total, bandwidth per day, number of >> writes per day, number of read per day. >> >> How can we calculate the number of writes and read per day? If I have a >> filled form with 50 answers that is uploaded to the server, that will be 50 >> writes? If I visualize results on Aggregate, how do they calculate the >> number of reads? >> >> Maxim >> >> >> >> On Wednesday, December 19, 2012 2:16:20 PM UTC-5, Maxim wrote: >>> >>> Hi, >>> >>> I am currently trying to better understand the billing system for a >>> future deployment using ODK on the Google AppEngine. The system would be >>> used as part of an environmental monitoring program for construction sites. >>> >>> Background information: >>> - 2 surveyors using Android phones >>> - 100 sites that would be visited over the course of 2 months >>> - 5 forms with an average of, let's say 20 questions, so a total of 100 >>> questions to be filled by site >>> - each entry would also have a low-resolution picture (about 200-300 kb >>> in size) >>> >>> I read the quotas and billing sections from the Help site but have some >>> trouble relating the information to what we are going to do. >>> >>> How can I use the background information to evaluate my future data >>> usage? I need to calculate the number of forms submitted per day, estimate >>> the total size of bandwidth used per day? I'm trying to see if I would be >>> OK with the minimum of 2.10$/week. >>> >>> Thanks, >>> >>> Maxim >>> >>> -- >> Post: opend...@googlegroups.com >> Unsubscribe: opendatakit...@googlegroups.com >> Options: http://groups.google.com/group/opendatakit?hl=en >> > > > > -- > Mitch Sundt > Software Engineer > University of Washington > mitche...@gmail.com

I've got several instances of aggregate, all with the $2.10 limit. Never
gone over even with some of our more robust forms.

··· On Dec 21, 2012 10:00 AM, "Mitch S" wrote:

The number of writes will generally not be a problem; you have 50,000 /
day for free.

The calculation for an individual submission is as follows:

  • one write for the submission,
  • one write for each filled-in repeat for every repeat group in that
    submission (if you have a household survey with 4 people (1st repeat
    group), and each person has 3 personal items (nested repeat group under
    'person'), then you would have 4 + 4x3 filled-in repeats -- for a total of
    16 writes),
  • one write for each selected item in any multiple-choice question,
  • 3 writes for each media (audio, video, image) attachment
  • 1 additional write for every 1,000,000 bytes of media attachment
    file size above the first 1,000,000 bytes.

For a very complex form with media, it is likely that you would have fewer
than 100 writes per submission. That would give you a daily budget of 500
submissions.

Not surprisingly, the number of required reads to access and display a
submission is the same as the number of writes.

But there are additional reads:

  • when you view the submissions, you are presented with 100
    submissions per page
  • the submissions pages are refreshed every 70 seconds (until you do
    nothing for 3 minutes, after which it will sleep); the other pages refresh
    more frequently.
  • the form definition must be read and cached (that cache expires
    every 3 seconds and the form definitions are re-read when next used). This
    is quite read-intensive, with one record per question and per group in the
    form.
  • the security information is re-read every 5 minutes to pick up the
    latest security settings changes
  • publishing requires reading each submission, and incurs 1 extra
    write per submission transmitted.
  • export to file requires reading each submission and incurs a handful
    of writes to save the formatted file.
  • a watchdog process kicks off every few minutes to detect stalled
    publishing or export jobs and restart them.

If you want to skim under the free limit, and find yourself hitting it,
you should minimize use of the website, and periodically do a pull of the
data using ODK Briefcase.

Mitch

On Fri, Dec 21, 2012 at 8:32 AM, Maxim maxfrom1987@gmail.com wrote:

Ok, thank you all for the additional information.

So it looks like we need to look at data usage using at least three
different levels: data storage space in total, bandwidth per day, number of
writes per day, number of read per day.

How can we calculate the number of writes and read per day? If I have a
filled form with 50 answers that is uploaded to the server, that will be 50
writes? If I visualize results on Aggregate, how do they calculate the
number of reads?

Maxim

On Wednesday, December 19, 2012 2:16:20 PM UTC-5, Maxim wrote:

Hi,

I am currently trying to better understand the billing system for a
future deployment using ODK on the Google AppEngine. The system would be
used as part of an environmental monitoring program for construction sites.

Background information:

  • 2 surveyors using Android phones
  • 100 sites that would be visited over the course of 2 months
  • 5 forms with an average of, let's say 20 questions, so a total of 100
    questions to be filled by site
  • each entry would also have a low-resolution picture (about 200-300 kb
    in size)

I read the quotas and billing sections from the Help site but have some
trouble relating the information to what we are going to do.

How can I use the background information to evaluate my future data
usage? I need to calculate the number of forms submitted per day, estimate
the total size of bandwidth used per day? I'm trying to see if I would be
OK with the minimum of 2.10$/week.

Thanks,

Maxim

--
Post: opendatakit@googlegroups.com
Unsubscribe: opendatakit+unsubscribe@googlegroups.com
Options: http://groups.google.com/group/opendatakit?hl=en

--
Mitch Sundt
Software Engineer
University of Washington
mitchellsundt@gmail.com

--
Post: opendatakit@googlegroups.com
Unsubscribe: opendatakit+unsubscribe@googlegroups.com
Options: http://groups.google.com/group/opendatakit?hl=en