Query regarding using ODK Collect as crowdsourcing tool

Support
1

Hello!
I have just discovered ODK and see it as a fantastic tool for collecting data. I plan to create a crowdsourcing application which will potentially be used by a large group of students to collect data. While I have figured out the server installation and data collection part, I have a few queries that I would love to pose to you:

  1. Instead of creating multiple accounts on 'Aggregate', can I let it authenticate the accounts using a 'Single Sign On' server without any major change to the Collect app? Or, at least, is there a way I can avoid creating the accounts manually on the Aggregate server by copying the credentials from some other database?

  2. I have disabled anonymous upload of data on my server. How can I design the form so that the username of the surveyor is also uploaded automatically (without him entering it in a field of the form) every time the form is submitted? This will help me identify the number of forms being uploaded by a particular user.

  3. It would be nice if the metadata somehow contained a unique device ID of the device being used to submit the forms.

I am sorry if this is a duplicate post. I tried searching for a relevant post but my google-fu seems to have failed me.

Cheers!

Gaurav

2

I was able to find the answer to my Queries 2 and 3 on the page
: https://opendatakit.org/help/form-design/examples/ under the section
"Property Values".

But the first query regarding the Single Sign On or bulk user account
creation remains unanswered. It would be very kind if someone could suggest
a solution.

Cheers!

Gaurav

··· On Friday, February 17, 2017 at 12:21:13 AM UTC+5:30, Gaurav Kumar wrote: > > Hello! > I have just discovered ODK and see it as a fantastic tool for collecting > data. I plan to create a crowdsourcing application which will potentially > be used by a large group of students to collect data. While I have figured > out the server installation and data collection part, I have a few queries > that I would love to pose to you: > > 1. Instead of creating multiple accounts on 'Aggregate', can I let it > authenticate the accounts using a 'Single Sign On' server without any major > change to the Collect app? Or, at least, is there a way I can avoid > creating the accounts manually on the Aggregate server by copying the > credentials from some other database? > > 2. I have disabled anonymous upload of data on my server. How can I design > the form so that the username of the surveyor is also uploaded > automatically (without him entering it in a field of the form) every time > the form is submitted? This will help me identify the number of forms being > uploaded by a particular user. > > 3. It would be nice if the metadata somehow contained a unique device ID > of the device being used to submit the forms. > > I am sorry if this is a duplicate post. I tried searching for a relevant > post but my google-fu seems to have failed me. > > Cheers! > > Gaurav
3

Data can be submitted to ODK Aggregate anonymously. This might be what you
want?

Beginning with ODK Aggregate 1.4.13, you can upload users and their
permissions in bulk.
However, for security reasons, you cannot set passwords on those users.
That is still a manual process.

image
image970×1269 101 KB

··· On Fri, Feb 17, 2017 at 1:04 AM, Gaurav Kumar wrote:

I was able to find the answer to my Queries 2 and 3 on the page :
https://opendatakit.org/help/form-design/examples/ under the section
"Property Values".

But the first query regarding the Single Sign On or bulk user account
creation remains unanswered. It would be very kind if someone could suggest
a solution.

Cheers!

Gaurav

On Friday, February 17, 2017 at 12:21:13 AM UTC+5:30, Gaurav Kumar wrote:

Hello!
I have just discovered ODK and see it as a fantastic tool for collecting
data. I plan to create a crowdsourcing application which will potentially
be used by a large group of students to collect data. While I have figured
out the server installation and data collection part, I have a few queries
that I would love to pose to you:

  1. Instead of creating multiple accounts on 'Aggregate', can I let it
    authenticate the accounts using a 'Single Sign On' server without any major
    change to the Collect app? Or, at least, is there a way I can avoid
    creating the accounts manually on the Aggregate server by copying the
    credentials from some other database?

  2. I have disabled anonymous upload of data on my server. How can I
    design the form so that the username of the surveyor is also uploaded
    automatically (without him entering it in a field of the form) every time
    the form is submitted? This will help me identify the number of forms being
    uploaded by a particular user.

  3. It would be nice if the metadata somehow contained a unique device ID
    of the device being used to submit the forms.

I am sorry if this is a duplicate post. I tried searching for a relevant
post but my google-fu seems to have failed me.

Cheers!

Gaurav

--

Post: opendatakit@googlegroups.com
Unsubscribe: opendatakit+unsubscribe@googlegroups.com
Options: http://groups.google.com/group/opendatakit?hl=en

You received this message because you are subscribed to the Google Groups
"ODK Community" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to opendatakit+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--
Mitch Sundt
Software Engineer
University of Washington
mitchellsundt@gmail.com

4

Thank you Mitch!
Your response helps to a certain extent, but it is not quite what I was
looking for.

We have another portal which is a group of a number of web applications
with almost common user base. The users can log into these individual
applications by creating credentials that are valid across these
applications. We have achieved this using Single Sign On or SSO
(https://en.wikipedia.org/wiki/Single_sign-on).

I was wondering if we could make use of the same user credentials by
integrating the same SSO for ODK Aggregate too.

But I am curious what can be achieved using this bulk upload of users. Will
they be able to log in and change their passwords?

Do you think my query will gain more traction if I post it as a separate
topic?

Thanks again!
Gaurav

··· On Friday, February 17, 2017 at 10:36:02 PM UTC+5:30, Mitch Sundt wrote: > > Data can be submitted to ODK Aggregate anonymously. This might be what you > want? > > Beginning with ODK Aggregate 1.4.13, you can upload users and their > permissions in bulk. > However, for security reasons, you cannot set passwords on those users. > That is still a manual process. > > > > On Fri, Feb 17, 2017 at 1:04 AM, Gaurav Kumar <gaura...@gmail.com > wrote: > >> I was able to find the answer to my Queries 2 and 3 on the page : >> https://opendatakit.org/help/form-design/examples/ under the section >> "Property Values". >> >> But the first query regarding the Single Sign On or bulk user account >> creation remains unanswered. It would be very kind if someone could suggest >> a solution. >> >> Cheers! >> >> Gaurav >> >> >> On Friday, February 17, 2017 at 12:21:13 AM UTC+5:30, Gaurav Kumar wrote: >>> >>> Hello! >>> I have just discovered ODK and see it as a fantastic tool for collecting >>> data. I plan to create a crowdsourcing application which will potentially >>> be used by a large group of students to collect data. While I have figured >>> out the server installation and data collection part, I have a few queries >>> that I would love to pose to you: >>> >>> 1. Instead of creating multiple accounts on 'Aggregate', can I let it >>> authenticate the accounts using a 'Single Sign On' server without any major >>> change to the Collect app? Or, at least, is there a way I can avoid >>> creating the accounts manually on the Aggregate server by copying the >>> credentials from some other database? >>> >>> 2. I have disabled anonymous upload of data on my server. How can I >>> design the form so that the username of the surveyor is also uploaded >>> automatically (without him entering it in a field of the form) every time >>> the form is submitted? This will help me identify the number of forms being >>> uploaded by a particular user. >>> >>> 3. It would be nice if the metadata somehow contained a unique device ID >>> of the device being used to submit the forms. >>> >>> I am sorry if this is a duplicate post. I tried searching for a relevant >>> post but my google-fu seems to have failed me. >>> >>> Cheers! >>> >>> Gaurav >> >> -- >> -- >> Post: opend...@googlegroups.com >> Unsubscribe: opendatakit...@googlegroups.com >> Options: http://groups.google.com/group/opendatakit?hl=en >> >> --- >> You received this message because you are subscribed to the Google Groups >> "ODK Community" group. >> To unsubscribe from this group and stop receiving emails from it, send an >> email to opendatakit...@googlegroups.com . >> For more options, visit https://groups.google.com/d/optout. >> > > > > -- > Mitch Sundt > Software Engineer > University of Washington > mitche...@gmail.com >
5

Gaurav,

Aggregate doesn't have SSO support, but it has some support for OAuth
2, so that'd be a good place to start if you want to contribute code
for LDAP backed SSO.

If you are looking for a quick hack, you might also be able to write
some code external to Aggregate to insert usernames/passwords into the
Aggregate DB.

I believe bulk upload lets you add a lot of user names. You then have
to set passwords for those users individually through the web UI. I
doubt this is what you are looking for.

Yaw

··· On Sat, Feb 18, 2017 at 9:11 PM, Gaurav Kumar wrote: > Thank you Mitch! > Your response helps to a certain extent, but it is not quite what I was > looking for. > > We have another portal which is a group of a number of web applications with > almost common user base. The users can log into these individual > applications by creating credentials that are valid across these > applications. We have achieved this using Single Sign On or SSO > (https://en.wikipedia.org/wiki/Single_sign-on). > > I was wondering if we could make use of the same user credentials by > integrating the same SSO for ODK Aggregate too. > > But I am curious what can be achieved using this bulk upload of users. Will > they be able to log in and change their passwords? > > Do you think my query will gain more traction if I post it as a separate > topic? > > Thanks again! > Gaurav > > On Friday, February 17, 2017 at 10:36:02 PM UTC+5:30, Mitch Sundt wrote: >> >> Data can be submitted to ODK Aggregate anonymously. This might be what you >> want? >> >> Beginning with ODK Aggregate 1.4.13, you can upload users and their >> permissions in bulk. >> However, for security reasons, you cannot set passwords on those users. >> That is still a manual process. >> >> >> >> On Fri, Feb 17, 2017 at 1:04 AM, Gaurav Kumar wrote: >>> >>> I was able to find the answer to my Queries 2 and 3 on the page : >>> https://opendatakit.org/help/form-design/examples/ under the section >>> "Property Values". >>> >>> But the first query regarding the Single Sign On or bulk user account >>> creation remains unanswered. It would be very kind if someone could suggest >>> a solution. >>> >>> Cheers! >>> >>> Gaurav >>> >>> >>> On Friday, February 17, 2017 at 12:21:13 AM UTC+5:30, Gaurav Kumar wrote: >>>> >>>> Hello! >>>> I have just discovered ODK and see it as a fantastic tool for collecting >>>> data. I plan to create a crowdsourcing application which will potentially be >>>> used by a large group of students to collect data. While I have figured out >>>> the server installation and data collection part, I have a few queries that >>>> I would love to pose to you: >>>> >>>> 1. Instead of creating multiple accounts on 'Aggregate', can I let it >>>> authenticate the accounts using a 'Single Sign On' server without any major >>>> change to the Collect app? Or, at least, is there a way I can avoid creating >>>> the accounts manually on the Aggregate server by copying the credentials >>>> from some other database? >>>> >>>> 2. I have disabled anonymous upload of data on my server. How can I >>>> design the form so that the username of the surveyor is also uploaded >>>> automatically (without him entering it in a field of the form) every time >>>> the form is submitted? This will help me identify the number of forms being >>>> uploaded by a particular user. >>>> >>>> 3. It would be nice if the metadata somehow contained a unique device ID >>>> of the device being used to submit the forms. >>>> >>>> I am sorry if this is a duplicate post. I tried searching for a relevant >>>> post but my google-fu seems to have failed me. >>>> >>>> Cheers! >>>> >>>> Gaurav >>> >>> -- >>> -- >>> Post: opend...@googlegroups.com >>> Unsubscribe: opendatakit...@googlegroups.com >>> Options: http://groups.google.com/group/opendatakit?hl=en >>> >>> --- >>> You received this message because you are subscribed to the Google Groups >>> "ODK Community" group. >>> To unsubscribe from this group and stop receiving emails from it, send an >>> email to opendatakit...@googlegroups.com. >>> For more options, visit https://groups.google.com/d/optout. >> >> >> >> >> -- >> Mitch Sundt >> Software Engineer >> University of Washington >> mitche...@gmail.com > > -- > -- > Post: opendatakit@googlegroups.com > Unsubscribe: opendatakit+unsubscribe@googlegroups.com > Options: http://groups.google.com/group/opendatakit?hl=en > > --- > You received this message because you are subscribed to the Google Groups > "ODK Community" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to opendatakit+unsubscribe@googlegroups.com. > For more options, visit https://groups.google.com/d/optout.
6

Thank you for the pointer Yaw! I shall look into OAuth. I think having a
SSO type authentication will enable ODK Collect being incorporated by
already established servers.

Cheers!
Gaurav

··· On Tuesday, February 21, 2017 at 5:51:33 AM UTC+5:30, Yaw Anokwa wrote: > > Gaurav, > > Aggregate doesn't have SSO support, but it has some support for OAuth > 2, so that'd be a good place to start if you want to contribute code > for LDAP backed SSO. > > If you are looking for a quick hack, you might also be able to write > some code external to Aggregate to insert usernames/passwords into the > Aggregate DB. > > I believe bulk upload lets you add a lot of user names. You then have > to set passwords for those users individually through the web UI. I > doubt this is what you are looking for. > > Yaw > > On Sat, Feb 18, 2017 at 9:11 PM, Gaurav Kumar <gaura...@gmail.com > wrote: > > Thank you Mitch! > > Your response helps to a certain extent, but it is not quite what I was > > looking for. > > > > We have another portal which is a group of a number of web applications > with > > almost common user base. The users can log into these individual > > applications by creating credentials that are valid across these > > applications. We have achieved this using Single Sign On or SSO > > (https://en.wikipedia.org/wiki/Single_sign-on). > > > > I was wondering if we could make use of the same user credentials by > > integrating the same SSO for ODK Aggregate too. > > > > But I am curious what can be achieved using this bulk upload of users. > Will > > they be able to log in and change their passwords? > > > > Do you think my query will gain more traction if I post it as a separate > > topic? > > > > Thanks again! > > Gaurav > > > > On Friday, February 17, 2017 at 10:36:02 PM UTC+5:30, Mitch Sundt wrote: > >> > >> Data can be submitted to ODK Aggregate anonymously. This might be what > you > >> want? > >> > >> Beginning with ODK Aggregate 1.4.13, you can upload users and their > >> permissions in bulk. > >> However, for security reasons, you cannot set passwords on those users. > >> That is still a manual process. > >> > >> > >> > >> On Fri, Feb 17, 2017 at 1:04 AM, Gaurav Kumar wrote: > >>> > >>> I was able to find the answer to my Queries 2 and 3 on the page : > >>> https://opendatakit.org/help/form-design/examples/ under the section > >>> "Property Values". > >>> > >>> But the first query regarding the Single Sign On or bulk user account > >>> creation remains unanswered. It would be very kind if someone could > suggest > >>> a solution. > >>> > >>> Cheers! > >>> > >>> Gaurav > >>> > >>> > >>> On Friday, February 17, 2017 at 12:21:13 AM UTC+5:30, Gaurav Kumar wrote: > >>>> > >>>> Hello! > >>>> I have just discovered ODK and see it as a fantastic tool for > collecting > >>>> data. I plan to create a crowdsourcing application which will > potentially be > >>>> used by a large group of students to collect data. While I have > figured out > >>>> the server installation and data collection part, I have a few > queries that > >>>> I would love to pose to you: > >>>> > >>>> 1. Instead of creating multiple accounts on 'Aggregate', can I let it > >>>> authenticate the accounts using a 'Single Sign On' server without any > major > >>>> change to the Collect app? Or, at least, is there a way I can avoid > creating > >>>> the accounts manually on the Aggregate server by copying the > credentials > >>>> from some other database? > >>>> > >>>> 2. I have disabled anonymous upload of data on my server. How can I > >>>> design the form so that the username of the surveyor is also uploaded > >>>> automatically (without him entering it in a field of the form) every > time > >>>> the form is submitted? This will help me identify the number of forms > being > >>>> uploaded by a particular user. > >>>> > >>>> 3. It would be nice if the metadata somehow contained a unique device > ID > >>>> of the device being used to submit the forms. > >>>> > >>>> I am sorry if this is a duplicate post. I tried searching for a > relevant > >>>> post but my google-fu seems to have failed me. > >>>> > >>>> Cheers! > >>>> > >>>> Gaurav > >>> > >>> -- > >>> -- > >>> Post: opend...@googlegroups.com > >>> Unsubscribe: opendatakit...@googlegroups.com > >>> Options: http://groups.google.com/group/opendatakit?hl=en > >>> > >>> --- > >>> You received this message because you are subscribed to the Google > Groups > >>> "ODK Community" group. > >>> To unsubscribe from this group and stop receiving emails from it, send > an > >>> email to opendatakit...@googlegroups.com. > >>> For more options, visit https://groups.google.com/d/optout. > >> > >> > >> > >> > >> -- > >> Mitch Sundt > >> Software Engineer > >> University of Washington > >> mitche...@gmail.com > > > > -- > > -- > > Post: opend...@googlegroups.com > > Unsubscribe: opendatakit...@googlegroups.com > > Options: http://groups.google.com/group/opendatakit?hl=en > > > > --- > > You received this message because you are subscribed to the Google > Groups > > "ODK Community" group. > > To unsubscribe from this group and stop receiving emails from it, send > an > > email to opendatakit...@googlegroups.com . > > For more options, visit https://groups.google.com/d/optout. >
7

Further questions should move to opendatakit-developers@, (added to this
response -- please remove opendatakit@ from any reply) as that is the
list for people modifying the software.

First, note that ODK Collect does not support anything other than BasicAuth
or DigestAuth.

The general architecture would insert a proxy webserver in front of ODK
Aggregate. That proxy webserver would implement your SSO mechanism and
would support BasicAuth. When using BasicAuth, ODK Collect enforces that
the connection MUST be https (must have an SSL certificate). This is
required because the username and password are effectively sent to the
server in plaintext when you use BasicAuth.

Assuming that your SSO mechanism can handle a BasicAuth handshake, then the
proxy server would guard/mediate/filter all requests to the underlying ODK
Aggregate server. For each request, a simple implementation would add a
session cookie that can be parsed within ODK Aggregate to extract that
username or, alternatively, transform the incoming request by adding an
additional header, X-Username with the username that the SSO mechanism has
verified.

The general approach to user and privilege management would be to configure
users and privileges through ODK Aggregate (e.g., bulk CSV upload), but do
not configure passwords -- i.e., let the proxy webserver manage passwords
and identity verification (authentication), and let ODK Aggregate manage
permissions assignments (authority). i.e., use the bulk update to
periodically update users at the appropriate privilege level into ODK
Aggregate or directly manipulate the database tables within ODK Aggregate
via database triggers to update in sync with your SSO user permissions
management service. e.g., _registered_users and _user_granted_authority

What remains is the modification of ODK Aggregate to retrieve and use the
out-of-band session cookie field or to parse and use the X-Username header
to identify the user making a request and enforce the authority
restrictions for that user.

This can be done by changing the functionality of the OutOfBand...
authentication classes.

An example of this is the now-obsolete Oauth 1.0 and IAM user
identification mechanism configured on Google AppEngine. This is wired up
for both GAE and for Tomcat, but here are the files for GAE e.g.,

and

https://github.com/opendatakit/aggregate/blob/master/war-base/WEB-INF/applicationContext-security.xml#L498

https://github.com/opendatakit/aggregate/blob/master/src/main/resources/gae/odk-settings.xml#L60

which reference:

https://github.com/opendatakit/aggregate/blob/master/src/main/java/org/opendatakit/common/utils/gae/GaeOutOfBandUserFetcher.java

The corresponding Tomcat class that you would then modify to tie into your
SSO solution is this:

https://github.com/opendatakit/aggregate/blob/master/src/main/java/org/opendatakit/common/utils/tomcat/TomcatOutOfBandUserFetcher.java

All that needs to be done is to change this last class to retrieve the user
from the request (e.g., X-Username header or from a session cookie managed
by the SSO layer).

Mitch

··· On Mon, Feb 20, 2017 at 8:42 PM, Gaurav Kumar wrote:

Thank you for the pointer Yaw! I shall look into OAuth. I think having a
SSO type authentication will enable ODK Collect being incorporated by
already established servers.

Cheers!
Gaurav

On Tuesday, February 21, 2017 at 5:51:33 AM UTC+5:30, Yaw Anokwa wrote:

Gaurav,

Aggregate doesn't have SSO support, but it has some support for OAuth
2, so that'd be a good place to start if you want to contribute code
for LDAP backed SSO.

If you are looking for a quick hack, you might also be able to write
some code external to Aggregate to insert usernames/passwords into the
Aggregate DB.

I believe bulk upload lets you add a lot of user names. You then have
to set passwords for those users individually through the web UI. I
doubt this is what you are looking for.

Yaw

On Sat, Feb 18, 2017 at 9:11 PM, Gaurav Kumar gaura...@gmail.com wrote:

Thank you Mitch!
Your response helps to a certain extent, but it is not quite what I was
looking for.

We have another portal which is a group of a number of web applications
with
almost common user base. The users can log into these individual
applications by creating credentials that are valid across these
applications. We have achieved this using Single Sign On or SSO
(https://en.wikipedia.org/wiki/Single_sign-on).

I was wondering if we could make use of the same user credentials by
integrating the same SSO for ODK Aggregate too.

But I am curious what can be achieved using this bulk upload of users.
Will
they be able to log in and change their passwords?

Do you think my query will gain more traction if I post it as a
separate
topic?

Thanks again!
Gaurav

On Friday, February 17, 2017 at 10:36:02 PM UTC+5:30, Mitch Sundt wrote:

Data can be submitted to ODK Aggregate anonymously. This might be what
you
want?

Beginning with ODK Aggregate 1.4.13, you can upload users and their
permissions in bulk.
However, for security reasons, you cannot set passwords on those
users.
That is still a manual process.

On Fri, Feb 17, 2017 at 1:04 AM, Gaurav Kumar gaura...@gmail.com wrote:

I was able to find the answer to my Queries 2 and 3 on the page :
https://opendatakit.org/help/form-design/examples/ under the section
"Property Values".

But the first query regarding the Single Sign On or bulk user account
creation remains unanswered. It would be very kind if someone could
suggest
a solution.

Cheers!

Gaurav

On Friday, February 17, 2017 at 12:21:13 AM UTC+5:30, Gaurav Kumar wrote:

Hello!
I have just discovered ODK and see it as a fantastic tool for
collecting
data. I plan to create a crowdsourcing application which will
potentially be
used by a large group of students to collect data. While I have
figured out
the server installation and data collection part, I have a few
queries that
I would love to pose to you:

  1. Instead of creating multiple accounts on 'Aggregate', can I let
    it
    authenticate the accounts using a 'Single Sign On' server without
    any major
    change to the Collect app? Or, at least, is there a way I can avoid
    creating
    the accounts manually on the Aggregate server by copying the
    credentials
    from some other database?

  2. I have disabled anonymous upload of data on my server. How can I
    design the form so that the username of the surveyor is also
    uploaded
    automatically (without him entering it in a field of the form) every
    time
    the form is submitted? This will help me identify the number of
    forms being
    uploaded by a particular user.

  3. It would be nice if the metadata somehow contained a unique
    device ID
    of the device being used to submit the forms.

I am sorry if this is a duplicate post. I tried searching for a
relevant
post but my google-fu seems to have failed me.

Cheers!

Gaurav

--

Post: opend...@googlegroups.com
Unsubscribe: opendatakit...@googlegroups.com
Options: http://groups.google.com/group/opendatakit?hl=en

You received this message because you are subscribed to the Google
Groups
"ODK Community" group.
To unsubscribe from this group and stop receiving emails from it,
send an
email to opendatakit...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--
Mitch Sundt
Software Engineer
University of Washington
mitche...@gmail.com

--

Post: opend...@googlegroups.com
Unsubscribe: opendatakit...@googlegroups.com
Options: http://groups.google.com/group/opendatakit?hl=en

You received this message because you are subscribed to the Google
Groups
"ODK Community" group.
To unsubscribe from this group and stop receiving emails from it, send
an
email to opendatakit...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--

Post: opendatakit@googlegroups.com
Unsubscribe: opendatakit+unsubscribe@googlegroups.com
Options: http://groups.google.com/group/opendatakit?hl=en

You received this message because you are subscribed to the Google Groups
"ODK Community" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to opendatakit+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--
Mitch Sundt
Software Engineer
University of Washington
mitchellsundt@gmail.com