I'm planning a survey in rural Africa using a questionnaire that will
result in around 400 variables (no pictures or complicated data, just
yes/no, select one, select multiple, dates questions). We are planning to
collect 2000 records over 7 days (using 75 data collectors, though I'm
guessing that is not so pertinent to this query). I've used ODK before, but
only when we had a dedicated database manager to handle the data. I
understand we can host the data using Google Appengine, and I've managed to
set up a practice database (for free), but I understand there are
limitations to how much data you can move through Appengine. would there be
any problems using appengine for a survey such as the one I propose? I know
that paying for appengine is also an option - would we need to pay for
space on the server with a survey such as this one?
Given the intense submission rate (300 submissions per day), you will
likely need to set up billing for AppEngine.
But the usage charges will be very small. When you set up billing, the
minimum daily charge is $0, so you don't incur a fee if you don't exceed
the free usage threshold. That would probably happen only during the data
collection phase and when you try to access all this data at once.
There are various options for accessing data
We recommend doing intensive data processing outside of ODK Aggregate. Once
you move the data to an analysis platform, you should be back below the
free usage threshold.
···
On Mon, Nov 16, 2015 at 10:48 AM, Colin Beckworth wrote:
Hi All,
I'm planning a survey in rural Africa using a questionnaire that will
result in around 400 variables (no pictures or complicated data, just
yes/no, select one, select multiple, dates questions). We are planning to
collect 2000 records over 7 days (using 75 data collectors, though I'm
guessing that is not so pertinent to this query). I've used ODK before, but
only when we had a dedicated database manager to handle the data. I
understand we can host the data using Google Appengine, and I've managed to
set up a practice database (for free), but I understand there are
limitations to how much data you can move through Appengine. would there be
any problems using appengine for a survey such as the one I propose? I know
that paying for appengine is also an option - would we need to pay for
space on the server with a survey such as this one?
Thanks Mitch, this is helpful information and much appreciated.
If I understand correctly I just link appengine to ODK aggregate as I have
done for free when practicing, but then set the account to billing and it
will allow as much data as required to flow in and out, applying charges
according to usage. You seem to think this will work fine for the amount of
data and usage I've proposed, and cost would be fairly minimal (so long as
its within a few hundred dollars it will be fine for this survey).
Yes, it should not be expensive. Probably under $20 US.
···
On Tue, Nov 17, 2015 at 8:54 AM, Colin Beckworth wrote:
Thanks Mitch, this is helpful information and much appreciated.
If I understand correctly I just link appengine to ODK aggregate as I have
done for free when practicing, but then set the account to billing and it
will allow as much data as required to flow in and out, applying charges
according to usage. You seem to think this will work fine for the amount of
data and usage I've proposed, and cost would be fairly minimal (so long as
its within a few hundred dollars it will be fine for this survey).
I can echo Mitch's estimate. I've completed many similar sized surveys
using app engine and have never exceeded $20 on a single deployment.
~lb
···
On Tuesday, November 17, 2015 at 2:10:26 PM UTC-5, Mitch Sundt wrote:
>
> Yes, it should not be expensive. Probably under $20 US.
>
> On Tue, Nov 17, 2015 at 8:54 AM, Colin Beckworth <cbeckw...@gmail.com > wrote:
>
>> Thanks Mitch, this is helpful information and much appreciated.
>>
>> If I understand correctly I just link appengine to ODK aggregate as I
>> have done for free when practicing, but then set the account to billing and
>> it will allow as much data as required to flow in and out, applying charges
>> according to usage. You seem to think this will work fine for the amount of
>> data and usage I've proposed, and cost would be fairly minimal (so long as
>> its within a few hundred dollars it will be fine for this survey).
>>
>> Thanks again,
>>
>> Colin
>>
>> --
>> --
>> Post: opend...@googlegroups.com
>> Unsubscribe: opendatakit...@googlegroups.com
>> Options: http://groups.google.com/group/opendatakit?hl=en
>>
>> ---
>> You received this message because you are subscribed to the Google Groups
>> "ODK Community" group.
>> To unsubscribe from this group and stop receiving emails from it, send an
>> email to opendatakit...@googlegroups.com .
>> For more options, visit https://groups.google.com/d/optout.
>>
>
>
>
> --
> Mitch Sundt
> Software Engineer
> University of Washington
> mitche...@gmail.com
>
Thanks Lloyd - can I also ask if you have ever had any issues using
Appengine with ODK? I have used ODK for other surveys, and really like it,
but only ever with a dedicated database run by a database manager. This is
the first time we will be managing the data ourselves, and I am concerned
that there will be issues that I haven't foreseen. I've set up ODK
aggregate using appengine myself before, but only in a practice run
entering one or two records. My background is in health rather than IT or
database management. Our survey will be in Chad.
For the organization I work for, TANGO International http://tangointernational.com/, we have four different ODK aggregate
servers deployed on appengine. I set them up, and I'm far from an IT
background (my background is applied economics). I've never had serious
(i.e. data loss) issues with them, and the few instances that I've had
minor questions, this forum has been a wealth of information.
I do suggest you run multiple 'test' deployments. By this I mean during
your training have enumerators collect practice data over the course of a
day, upload the data, you download the data, and convert it to the format
you expect to do analysis in. Similarly, before you ever make it to the
training stage, you'll want to run similar test deployments with yourself
and fellow colleagues till you are comfortable with the process.
Since you will be deploying in Chad, I also suggest the survey manager and
trainer become comfortable working in an offline setting (i.e. know how
proceed with fieldwork offline if needed -- briefcase is a wonder in
supporting this).
Good luck.
···
On Wednesday, November 18, 2015 at 11:35:15 AM UTC-5, Colin Beckworth wrote:
>
>
> Thanks Lloyd - can I also ask if you have ever had any issues using
> Appengine with ODK? I have used ODK for other surveys, and really like it,
> but only ever with a dedicated database run by a database manager. This is
> the first time we will be managing the data ourselves, and I am concerned
> that there will be issues that I haven't foreseen. I've set up ODK
> aggregate using appengine myself before, but only in a practice run
> entering one or two records. My background is in health rather than IT or
> database management. Our survey will be in Chad.
>
> Thanks,
>
> Colin
>
Thanks very much for this, its very reassuring - we plan to do lots of
testing a month before live data collection, exactly as you suggest, but
because I'm not an IT guy, I'm afraid of the 'things I don't know that that
I don't know'. However, it sounds from your experience like it works well
live provided it is thoroughly tested pre-deployment.
Thanks again, agreed this forum is fantastic resource.
I'm about to launch a similar study (a little smaller) and was also wondering about appengine vs dedicated server. Sounds like appengine is the way to go! And so cheap! Great to hear.
Given that I go with appengine, I was wondering what the suggested backup method would be. I gather that google offers one option (https://cloud.google.com/appengine/docs/developers-console/datastoreadmin?hl=en#backuprestore_copy_and_delete_data). Another option might be to write a cronjob on a remote server that uses Briefcase's command line interface to do periodic exports (https://opendatakit.org/use/briefcase/). The latter seems a little hacky. The former a little over the top (esp given that what we'd really need is also periodic exports instead/in addition to backups... for data analysis purposes).
Having said all that... I'd love to hear what other people suggest here.
I'm big fan of the cron Briefcase CLI as backup. It's easy to automate
and store on other systems (S3, Dropbox) and easy to restore to
another server.
Yaw
···
--
Need ODK consultants? Nafundi provides form design, server setup,
in-field training, and software development for ODK. Go to
https://nafundi.com to get started.
I'm about to launch a similar study (a little smaller) and was also wondering about appengine vs dedicated server. Sounds like appengine is the way to go! And so cheap! Great to hear.
Given that I go with appengine, I was wondering what the suggested backup method would be. I gather that google offers one option (https://cloud.google.com/appengine/docs/developers-console/datastoreadmin?hl=en#backuprestore_copy_and_delete_data). Another option might be to write a cronjob on a remote server that uses Briefcase's command line interface to do periodic exports (https://opendatakit.org/use/briefcase/). The latter seems a little hacky. The former a little over the top (esp given that what we'd really need is also periodic exports instead/in addition to backups... for data analysis purposes).
Having said all that... I'd love to hear what other people suggest here.
While Windows and Apple products wrap and give you complete solutions to do
X, the earlier mainframe operating systems, and others like Linux, are all
about being given a set of simple small tools that do only one or two small
things, none of which individually accomplish X.
And then users are expected to know enough of these tools to be able to
compose them into a solution that does X.
In a web context, this is called a 'mashup'
But its been around since at least JCL.
···
On Tue, Nov 24, 2015 at 8:26 PM, Yaw Anokwa wrote:
I'm big fan of the cron Briefcase CLI as backup. It's easy to automate
and store on other systems (S3, Dropbox) and easy to restore to
another server.
Yaw
Need ODK consultants? Nafundi provides form design, server setup,
in-field training, and software development for ODK. Go to https://nafundi.com to get started.
I'm about to launch a similar study (a little smaller) and was also
wondering about appengine vs dedicated server. Sounds like appengine is the
way to go! And so cheap! Great to hear.
Given that I go with appengine, I was wondering what the suggested
backup method would be. I gather that google offers one option ( https://cloud.google.com/appengine/docs/developers-console/datastoreadmin?hl=en#backuprestore_copy_and_delete_data).
Another option might be to write a cronjob on a remote server that uses
Briefcase's command line interface to do periodic exports ( https://opendatakit.org/use/briefcase/). The latter seems a little hacky.
The former a little over the top (esp given that what we'd really need is
also periodic exports instead/in addition to backups... for data analysis
purposes).
Having said all that... I'd love to hear what other people suggest here.
Sweet. I kinda wanted to do it that way anyways, since it meets all the goals: cheap, lightweight, produces exports... and a little bit of learning tossed in.
Thanks for the input
···
On Wednesday, November 25, 2015 at 12:27:03 PM UTC-5, Mitch Sundt wrote:
> I wouldn't say that:
>
>
> cronjob + Briefcase
>
>
> is hacky.
>
>
> While Windows and Apple products wrap and give you complete solutions to do X, the earlier mainframe operating systems, and others like Linux, are all about being given a set of simple small tools that do only one or two small things, none of which individually accomplish X.
>
>
> And then users are expected to know enough of these tools to be able to compose them into a solution that does X.
>
>
> In a web context, this is called a 'mashup'
>
>
> But its been around since at least JCL.
>
>
>
>
>
>
>
>
>
>
>
>
> On Tue, Nov 24, 2015 at 8:26 PM, Yaw Anokwa wrote:
> I'm big fan of the cron Briefcase CLI as backup. It's easy to automate
>
> and store on other systems (S3, Dropbox) and easy to restore to
>
> another server.
>
>
>
> Yaw
>
> --
>
> Need ODK consultants? Nafundi provides form design, server setup,
>
> in-field training, and software development for ODK. Go to
>
> https://nafundi.com to get started.
>
>
>
>
>
> On Wed, Nov 25, 2015 at 11:14 AM, wrote:
>
> > Hi all,
>
> >
>
> > I'm about to launch a similar study (a little smaller) and was also wondering about appengine vs dedicated server. Sounds like appengine is the way to go! And so cheap! Great to hear.
>
> >
>
> > Given that I go with appengine, I was wondering what the suggested backup method would be. I gather that google offers one option (https://cloud.google.com/appengine/docs/developers-console/datastoreadmin?hl=en#backuprestore_copy_and_delete_data). Another option might be to write a cronjob on a remote server that uses Briefcase's command line interface to do periodic exports (https://opendatakit.org/use/briefcase/). The latter seems a little hacky. The former a little over the top (esp given that what we'd really need is also periodic *exports* instead/in addition to backups... for data analysis purposes).
>
> >
>
> > Having said all that... I'd love to hear what other people suggest here.
>
> >
>
> > Thanks!
>
> >
>
> > --
>
> > --
>
> > Post: opend...@googlegroups.com
>
> > Unsubscribe: opendatakit...@googlegroups.com
>
> > Options: http://groups.google.com/group/opendatakit?hl=en
>
> >
>
> > ---
>
> > You received this message because you are subscribed to the Google Groups "ODK Community" group.
>
> > To unsubscribe from this group and stop receiving emails from it, send an email to opendatakit...@googlegroups.com.
>
> > For more options, visit https://groups.google.com/d/optout.
>
>
>
> --
>
> --
>
> Post: opend...@googlegroups.com
>
> Unsubscribe: opendatakit...@googlegroups.com
>
> Options: http://groups.google.com/group/opendatakit?hl=en
>
>
>
> ---
>
> You received this message because you are subscribed to the Google Groups "ODK Community" group.
>
> To unsubscribe from this group and stop receiving emails from it, send an email to opendatakit...@googlegroups.com.
>
> For more options, visit https://groups.google.com/d/optout.
>
>
>
>
>
> --
>
> Mitch Sundt
> Software Engineer
> University of Washington
> mitche...@gmail.com