Hello,
I am eagerly waiting for the ODK Collect v1.1.8 release for this
feature - "Encryption of data on phone (with user-defined PIN) and to/
on/from server (with public key)"
Any idea when the release is expected to happen? 
Cheers
Srini
we are trying for the end of the year, but don't count on it.
···
On Wed, Nov 9, 2011 at 08:22, Srini wrote: > Hello, > > I am eagerly waiting for the ODK Collect v1.1.8 release for this > feature - "Encryption of data on phone (with user-defined PIN) and to/ > on/from server (with public key)" > > Any idea when the release is expected to happen? :) > > Cheers > SriniAny other thoughts on encryption?
I have been looking at eMocha's version of ODK, but that solution requires
giving up discrete XML file instances and writing everything into the
onboard sql database.
Has anyone tried working with the Android Guardian project? Other options
for encrypting the SDcard in its entirety?
☞§※☼:airplane:
~Neil
···
On Wed, Nov 9, 2011 at 12:27 AM, Yaw Anokwa wrote:we are trying for the end of the year, but don't count on it.
On Wed, Nov 9, 2011 at 08:22, Srini vsrinivasan3@gmail.com wrote:
Hello,
I am eagerly waiting for the ODK Collect v1.1.8 release for this
feature - "Encryption of data on phone (with user-defined PIN) and to/
on/from server (with public key)"Any idea when the release is expected to happen?
Cheers
Srini
here are some (half-baked) thoughts...
i don't think emocha encrypts the data. they write it to the internal
sdcard's db. this is ok security as long as the phone isn't rooted.
when i tried the android guardian demo it didn't work well, so caveat
emptor. it also puts everything in db, which we may not want to do.
android 4.0 has support for encrypting the entire device. that's not
ideal because we want to support encryption on existing devices.
dimagi has some encryption code in commcare-odk. don't know exactly
how it works and the limitations, but it is likely a good place to
start looking if you want to encrypt instances. taking a closer look
is on my todo list.
the ideal encryption implementation supports all devices, allows for
encryption without databases, and provides very strong encryption
without long passwords. we think we have a good first pass that will
work better for our users than the stuff that's out there now...
···
On Thursday, November 10, 2011, Neil Hendrick wrote: > > Any other thoughts on encryption? > I have been looking at eMocha's version of ODK, but that solution requires giving up discrete XML file instances and writing everything into the onboard sql database. > Has anyone tried working with the Android Guardian project? Other options for encrypting the SDcard in its entirety? > > ☞§※☼✈☂☻ > ~Neil > > > On Wed, Nov 9, 2011 at 12:27 AM, Yaw Anokwa wrote: >> >> we are trying for the end of the year, but don't count on it. >> >> On Wed, Nov 9, 2011 at 08:22, Srini wrote: >> > Hello, >> > >> > I am eagerly waiting for the ODK Collect v1.1.8 release for this >> > feature - "Encryption of data on phone (with user-defined PIN) and to/ >> > on/from server (with public key)" >> > >> > Any idea when the release is expected to happen? :) >> > >> > Cheers >> > Srini >Neil:
the timeline on having basic encryption (encrypted data and media
after the form is marked 'complete' and therefore no longer editable
on the client), support for storing data in Aggregate, and download
and decryption on a local computer (via Briefcase) will be coming by
year end. We will not be encrypting 'in progress' forms so they can
continue to be edited. Do you think it reasonable to make completed
forms no longer editable or should we wait to encrypt until the data
is uploaded? Of course, that gets us into other issues if the data is
moved off the phone with non-ODK tools.
Gaetano
···
On Wed, Nov 9, 2011 at 8:11 PM, Yaw Anokwa wrote: > here are some (half-baked) thoughts... > > i don't think emocha encrypts the data. they write it to the internal > sdcard's db. this is ok security as long as the phone isn't rooted. > > when i tried the android guardian demo it didn't work well, so caveat > emptor. it also puts everything in db, which we may not want to do. > > android 4.0 has support for encrypting the entire device. that's not > ideal because we want to support encryption on existing devices. > > dimagi has some encryption code in commcare-odk. don't know exactly > how it works and the limitations, but it is likely a good place to > start looking if you want to encrypt instances. taking a closer look > is on my todo list. > > the ideal encryption implementation supports all devices, allows for > encryption without databases, and provides very strong encryption > without long passwords. we think we have a good first pass that will > work better for our users than the stuff that's out there now... > > > On Thursday, November 10, 2011, Neil Hendrick wrote: >> >> Any other thoughts on encryption? >> I have been looking at eMocha's version of ODK, but that solution requires giving up discrete XML file instances and writing everything into the onboard sql database. >> Has anyone tried working with the Android Guardian project? Other options for encrypting the SDcard in its entirety? >> >> ☞§※☼✈☂☻ >> ~Neil >> >> >> On Wed, Nov 9, 2011 at 12:27 AM, Yaw Anokwa wrote: >>> >>> we are trying for the end of the year, but don't count on it. >>> >>> On Wed, Nov 9, 2011 at 08:22, Srini wrote: >>> > Hello, >>> > >>> > I am eagerly waiting for the ODK Collect v1.1.8 release for this >>> > feature - "Encryption of data on phone (with user-defined PIN) and to/ >>> > on/from server (with public key)" >>> > >>> > Any idea when the release is expected to happen? :) >>> > >>> > Cheers >>> > Srini >> >for forms that are in-progress, we could encrypt with weaker 4-8 digit
pin. once the form is completed, then we encrypt with a strong public
key that is specified in the form.
···
On Thu, Nov 10, 2011 at 07:31, Gaetano Borriello wrote: > Neil: > > the timeline on having basic encryption (encrypted data and media > after the form is marked 'complete' and therefore no longer editable > on the client), support for storing data in Aggregate, and download > and decryption on a local computer (via Briefcase) will be coming by > year end. We will not be encrypting 'in progress' forms so they can > continue to be edited. Do you think it reasonable to make completed > forms no longer editable or should we wait to encrypt until the data > is uploaded? Of course, that gets us into other issues if the data is > moved off the phone with non-ODK tools. > > Gaetano > > On Wed, Nov 9, 2011 at 8:11 PM, Yaw Anokwa wrote: >> here are some (half-baked) thoughts... >> >> i don't think emocha encrypts the data. they write it to the internal >> sdcard's db. this is ok security as long as the phone isn't rooted. >> >> when i tried the android guardian demo it didn't work well, so caveat >> emptor. it also puts everything in db, which we may not want to do. >> >> android 4.0 has support for encrypting the entire device. that's not >> ideal because we want to support encryption on existing devices. >> >> dimagi has some encryption code in commcare-odk. don't know exactly >> how it works and the limitations, but it is likely a good place to >> start looking if you want to encrypt instances. taking a closer look >> is on my todo list. >> >> the ideal encryption implementation supports all devices, allows for >> encryption without databases, and provides very strong encryption >> without long passwords. we think we have a good first pass that will >> work better for our users than the stuff that's out there now... >> >> >> On Thursday, November 10, 2011, Neil Hendrick wrote: >>> >>> Any other thoughts on encryption? >>> I have been looking at eMocha's version of ODK, but that solution requires giving up discrete XML file instances and writing everything into the onboard sql database. >>> Has anyone tried working with the Android Guardian project? Other options for encrypting the SDcard in its entirety? >>> >>> ☞§※☼✈☂☻ >>> ~Neil >>> >>> >>> On Wed, Nov 9, 2011 at 12:27 AM, Yaw Anokwa wrote: >>>> >>>> we are trying for the end of the year, but don't count on it. >>>> >>>> On Wed, Nov 9, 2011 at 08:22, Srini wrote: >>>> > Hello, >>>> > >>>> > I am eagerly waiting for the ODK Collect v1.1.8 release for this >>>> > feature - "Encryption of data on phone (with user-defined PIN) and to/ >>>> > on/from server (with public key)" >>>> > >>>> > Any idea when the release is expected to happen? :) >>>> > >>>> > Cheers >>>> > Srini >>> >> >The Guardian Project is about to release SQLCipher 1.0 final and I have already begun implementing support for it in ODK Collect providers. I can post my branch in a few days. I have a good portion done in just a few hours of work because our database API is nearly identical to android.database.*.
Storage of files in the clear on the SD card is not just a root issue. Any app can read any data stored on the external storage partition. There are no special permission controls. This means any app can read, write, manipulate or corrupt the ODK data. Safe, reliable storage and verification of data should be considered a basic requirement and not an advanced feature.
Regarding the use of files vs. databases to store data - it would be helpful for me to understand why there is a need to use files? You can still use a document XML model, but just store that data as SQLite blobs. The way our SQLCipher layer works is that no data is every stored in a decrypted state on the file system.
As I said, we have great interest in showing how our upcoming release can be easily implemented as a core feature of ODK Collect. Any guidance that can be provided is much appreciated.
Best,
Nathan of Guardian
here are some (half-baked) thoughts...
i don't think emocha encrypts the data. they write it to the internal
sdcard's db. this is ok security as long as the phone isn't rooted.
when i tried the android guardian demo it didn't work well, so caveat
emptor. it also puts everything in db, which we may not want to do.
android 4.0 has support for encrypting the entire device. that's not
ideal because we want to support encryption on existing devices.
dimagi has some encryption code in commcare-odk. don't know exactly
how it works and the limitations, but it is likely a good place to
start looking if you want to encrypt instances. taking a closer look
is on my todo list.
the ideal encryption implementation supports all devices, allows for
encryption without databases, and provides very strong encryption
without long passwords. we think we have a good first pass that will
work better for our users than the stuff that's out there now...
···
Yaw Anokwa wrote:On Thursday, November 10, 2011, Neil Hendrick wrote:
Any other thoughts on encryption?
I have been looking at eMocha's version of ODK, but that solution requires giving up discrete XML file instances and writing everything into the onboard sql database.
Has anyone tried working with the Android Guardian project? Other options for encrypting the SDcard in its entirety?☞§※☼:airplane:
~NeilOn Wed, Nov 9, 2011 at 12:27 AM, Yaw Anokwa yanokwa@gmail.com wrote:
we are trying for the end of the year, but don't count on it.
On Wed, Nov 9, 2011 at 08:22, Srini vsrinivasan3@gmail.com wrote:
Hello,
I am eagerly waiting for the ODK Collect v1.1.8 release for this
feature - "Encryption of data on phone (with user-defined PIN) and to/
on/from server (with public key)"Any idea when the release is expected to happen?
Cheers
Srini
Your "half-baked thoughts" are well on the way to a tasty pastry of
security, to which I might add here, a few sprinkles of yeast to help rise
the dough, and a soupçon of butter to ensure a golden crispy crust.
As you say, "the ideal encryption implementation supports all devices,
allows for
encryption without databases, and provides very strong encryption
without long passwords."
All agreed, and I take Gaetano's point regarding finished vs. unfinished
forms. To my mind, an unfinished form is not worth returning to, and ALL
forms, no matter how incomplete, are finished forms. Therefore, I don't see
any issue with locking out any form which has been opened, edited, and
exited. Should a user wish to add more data, user can open a new form. Some
may have thoughts to the contrary. Yaw's compromise with lesser security
for incomplete forms will likely be most popular in that it allows for
returns and corrections, in which some users may see utility.
I think the point that should be made is that the data should be secure* in
transfer,* when posted to Aggregate, and when stored on the Aggregate
server. For those not using the Aggregate server solution, data must be
secure both in storage on the SDcard as instances, and there must be a
mechanism for accessing those instances and decrypting them when
transferred to another storage medium (viz. user's laptop or desktop)
So the question follows of at what level does the data become encrypted?
Example: My data is in an instance, which is an xml file called *
survey11102011.xml* which contains the following:
so, the data could be handled several ways, not sure how your solution in
progress handles this.
- instances folder could be encrypted so that there is nothing
recognizable there. - instances folder could contain XML files with meaninless names:
awfljj34narfnl.xml and content unreadable - instances folder contains instances *survey11102011.xml,*etc but the
data within is unreadable. - *survey11102011.xml *contains a recognizeable data structure, but the
answers themselves are unreadable:
That said, I imagine you have already settled on a scheme, I wonder if you
could further illuminate the intended method of protecting data.
☞§※☼:airplane:
~Neil
Nathan,
"Regarding the use of files vs. databases to store data"
the answer is really that files are easy. You can see them and manipulate
them using drag and drop or the Debug Bridge.
That said, your point about the security or its lack on the SDcard ( "Any
app can read any data stored on the external storage partition.") is very
well taken.
I'd like to see you accomodate ease of synchronization, users need to be
able to move data easily from SQLite blobs back to XML files, synch by
cable to a laptop or sync over the air to Aggregate or Dropbox.
We're looking forward to your Guardian implementation, I would love to see
a more secure method for storing completed surveys and will be testing it
as soon as it's available.
☞§※☼:airplane:
~Neil
···
On Fri, Nov 11, 2011 at 2:11 AM, Nathan Freitas wrote:Regarding the use of files vs. databases to store data
hilarious opening, neil
we will be encrypting the entire instance file (#3 in your list) with
a public key that will be embedded in the form definition.
when you send the encrypted instance to aggregate, it will be sent
over a secure channel (not that it matters, since it's already
encrypted), and stored as a encrypted blob. no one (including the
server owners) will be able to read the blob. the limitation here is
that you won't be able to do anything online (no viewing csv, maps,
graphs, etc).
to view the data, you will use briefcase to download the blob, enter
your private key there, and then data will be decrypted on your local
machine where you can do with it as you wish. if you don't have a
server, then briefcase will pull the data
from the phone, and decrypt it as if it came from the server.
i don't think we'll be encrypting the blank form, but that seems like
a pretty good idea (maybe with a weaker pin that user specifies and
enters time they launch the form). if we do this, then it's probably a
good idea to obfuscate the form and instance name.
···
On Fri, Nov 11, 2011 at 02:01, Neil Hendrick wrote: > Your "half-baked thoughts" are well on the way to a tasty pastry of > security, to which I might add here, a few sprinkles of yeast to help rise > the dough, and a soupçon of butter to ensure a golden crispy crust. > As you say, "the ideal encryption implementation supports all devices, > allows for > encryption without databases, and provides very strong encryption > without long passwords." > All agreed, and I take Gaetano's point regarding finished vs. unfinished > forms. To my mind, an unfinished form is not worth returning to, and ALL > forms, no matter how incomplete, are finished forms. Therefore, I don't see > any issue with locking out any form which has been opened, edited, and > exited. Should a user wish to add more data, user can open a new form. Some > may have thoughts to the contrary. Yaw's compromise with lesser security for > incomplete forms will likely be most popular in that it allows for returns > and corrections, in which some users may see utility. > I think the point that should be made is that the data should be secure in > transfer, when posted to Aggregate, and when stored on the Aggregate server. > For those not using the Aggregate server solution, data must be secure both > in storage on the SDcard as instances, and there must be a mechanism for > accessing those instances and decrypting them when transferred to another > storage medium (viz. user's laptop or desktop) > So the question follows of at what level does the data become encrypted? > Example: My data is in an instance, which is an xml file called > survey11102011.xml which contains the following: > > > > neil > 41 > Tampa > > > > so, the data could be handled several ways, not sure how your solution in > progress handles this. > > instances folder could be encrypted so that there is > nothing recognizable there. > instances folder could contain XML files with meaninless names: > awfljj34narfnl.xml and content unreadable > instances folder contains instances survey11102011.xml,etc but the data > within is unreadable. > survey11102011.xml contains a recognizeable data structure, but the answers > themselves are unreadable: > > > > aw;lfr > sf09dfav > aslckral > > > > That said, I imagine you have already settled on a scheme, I wonder if you > could further illuminate the intended method of protecting data. > ☞§※☼✈☂☻ > ~Neil > >#3 is a good option, and thanks for the explanation.
I know development is not on a fixed schedule, but what is the estimated
release date for this encryption function?
☞§※☼:airplane:
~Neil
Unfortunately with the changes to App Engine billing we have had to
scramble this week to make changes to Aggregate so overall our
development schedule got messed up. Over half of the ODK core team is
traveling or in the field right now so we haven't produced a new
schedule yet and the whole app engine situation is still unresolved. I
think before the end of the year is a good estimate as that is about
1.5 months away. We maybe able to have an estimated release date in a
week or two when everyone returns, but with all the change that
happened this week I think we are all a little hesitant to say when
until a few things other things are nailed down.
Waylon