Sequential Numbering in ODK Forms - Newbie

Hi,

I just started building a form for data collection. I'm most familiar with
using forms in ArcPAD so this is a little new for me. I'm wanting to
generate a sequential number for each new record. Is there an simple way of
having the form add 1 to the previous record? I've done this in ArcPAD but
couldn't find much info here or on simple google searches.

Thanks,
Heath

Not without some custom code. Collect will automatically provide a
unique identifier for a survey instance but not a sequential one.

The problem is how to keep sequential numbers across multiple phones
accurate as what does it mean if someone survey's person 1 on phone 1
and then survey with person 1 on phone 2? The numbers won't be
sequential. It becomes application dependent on how you want to handle
sequential numbers and what does it mean to have a sequential number
does it refer to the phone? a person being interviewed? time of day?
what happens when there are multiple phones and their sequence numbers
collide? While there are solutions to this problem we have not found a
good common solution that works for most use cases. People seem to
diverge in how things should be handled so we have not added this
functionality in yet to the common core and instead let people
implement their definition of sequential to the code base.

If you do not have the technical ability in your organization to make
the necessary change may I suggest looking at the companies that are
willing to be hired to help on our website.

We plan to add something to make this easier in ODK 2.0 but that is
still months off from being ready.

There has been some previous discussion on the forum before on this:
https://groups.google.com/forum/#!msg/opendatakit/uXaz3Ku97UU/8mDu606rJ7MJ

··· On Wed, Oct 24, 2012 at 10:58 PM, Heath Smith wrote: > Hi, > > I just started building a form for data collection. I'm most familiar with > using forms in ArcPAD so this is a little new for me. I'm wanting to > generate a sequential number for each new record. Is there an simple way of > having the form add 1 to the previous record? I've done this in ArcPAD but > couldn't find much info here or on simple google searches. > > Thanks, > Heath > > -- > Post: opendatakit@googlegroups.com > Unsubscribe: opendatakit+unsubscribe@googlegroups.com > Options: http://groups.google.com/group/opendatakit?hl=en

I think, in the ODK convention, a "record" refers to the whole dataset
captured in a form. 1 form entry = 1 record. Trying to number these records
consecutively will be an exercise in futility since a form wouldn't have an
idea how many other forms out there are collecting records.

Now, if you're collecting a list of items within the one form, you can
number these each consecutively but once you finalize and save the form,
your numbers reset back to the start.

So, which one is your use case?

/r
Nik

··· On Thursday, October 25, 2012, Heath Smith wrote:

Hi,

I just started building a form for data collection. I'm most familiar with
using forms in ArcPAD so this is a little new for me. I'm wanting to
generate a sequential number for each new record. Is there an simple way of
having the form add 1 to the previous record? I've done this in ArcPAD but
couldn't find much info here or on simple google searches.

Thanks,
Heath

--
Post: opendatakit@googlegroups.com <javascript:_e({}, 'cvml',
'opendatakit@googlegroups.com');>
Unsubscribe: opendatakit+unsubscribe@googlegroups.com <javascript:_e({},
'cvml', 'opendatakit%2Bunsubscribe@googlegroups.com');>
Options: http://groups.google.com/group/opendatakit?hl=en

As a side note the this, if we wanted to add sequential numbers to
submissions based on sequence forms have been submitted to agregate, would
this be possible?

I.E. Each new record in ODK aggregate gets a sequential number on
submission?

··· On Thursday, 25 October 2012 07:58:13 UTC+2, Heath Smith wrote: > > Hi, > > I just started building a form for data collection. I'm most familiar with > using forms in ArcPAD so this is a little new for me. I'm wanting to > generate a sequential number for each new record. Is there an simple way of > having the form add 1 to the previous record? I've done this in ArcPAD but > couldn't find much info here or on simple google searches. > > Thanks, > Heath >

Thanks Waylon,

Sounds like we'll have to wait a bit. Just to give you a bit of extra info.
We'd be collecting a location on 6 various phones. Each phone would be
dedicated to a team and their team name would be recorded with each
location. Thereby eliminating any confusion with multiple numbers showing
up in the database. With field collection it is often hard for technicians
to recall the last number they were at, hence the importance of the number
generator to mark each occurrence.

I checked out the link you provided to the thread but didn't find much
extra information. I saw where we can modify the code but that seems a
little over our heads here.

Thanks again for the quick response,
Heath

··· On Wednesday, October 24, 2012 11:20:20 PM UTC-7, Waylon Brunette wrote: > > Not without some custom code. Collect will automatically provide a > unique identifier for a survey instance but not a sequential one. > > The problem is how to keep sequential numbers across multiple phones > accurate as what does it mean if someone survey's person 1 on phone 1 > and then survey with person 1 on phone 2? The numbers won't be > sequential. It becomes application dependent on how you want to handle > sequential numbers and what does it mean to have a sequential number > does it refer to the phone? a person being interviewed? time of day? > what happens when there are multiple phones and their sequence numbers > collide? While there are solutions to this problem we have not found a > good common solution that works for most use cases. People seem to > diverge in how things should be handled so we have not added this > functionality in yet to the common core and instead let people > implement their definition of sequential to the code base. > > If you do not have the technical ability in your organization to make > the necessary change may I suggest looking at the companies that are > willing to be hired to help on our website. > http://opendatakit.org/help/help-for-hire/ > > We plan to add something to make this easier in ODK 2.0 but that is > still months off from being ready. > > There has been some previous discussion on the forum before on this: > https://groups.google.com/forum/#!msg/opendatakit/uXaz3Ku97UU/8mDu606rJ7MJ > > On Wed, Oct 24, 2012 at 10:58 PM, Heath Smith <heat...@gmail.com> wrote: > > Hi, > > > > I just started building a form for data collection. I'm most familiar > with > > using forms in ArcPAD so this is a little new for me. I'm wanting to > > generate a sequential number for each new record. Is there an simple way > of > > having the form add 1 to the previous record? I've done this in ArcPAD > but > > couldn't find much info here or on simple google searches. > > > > Thanks, > > Heath > > > > -- > > Post: opend...@googlegroups.com > > Unsubscribe: opendatakit...@googlegroups.com > > Options: http://groups.google.com/group/opendatakit?hl=en >

I'm use to collecting data on ArcPAD in which the database is localized on
the machine till we download it. We won't have cell service at our field
site so each phone will serve as a single database until we download it at
camp. My hope was that the form could search the phone for the last record
and add 1 to that records number, giving the new record the next number. It
would matter to us that other phones will have the same record on them
because when we download it we'll be able to see which phone that record
came from.

H

··· On Wednesday, October 24, 2012 11:35:31 PM UTC-7, ニコノコ wrote: > > I think, in the ODK convention, a "record" refers to the whole dataset > captured in a form. 1 form entry = 1 record. Trying to number these records > consecutively will be an exercise in futility since a form wouldn't have an > idea how many other forms out there are collecting records. > > Now, if you're collecting a list of items within the one form, you can > number these each consecutively but once you finalize and save the form, > your numbers reset back to the start. > > So, which one is your use case? > > /r > Nik > > On Thursday, October 25, 2012, Heath Smith wrote: > >> Hi, >> >> I just started building a form for data collection. I'm most familiar >> with using forms in ArcPAD so this is a little new for me. I'm wanting to >> generate a sequential number for each new record. Is there an simple way of >> having the form add 1 to the previous record? I've done this in ArcPAD but >> couldn't find much info here or on simple google searches. >> >> Thanks, >> Heath >> >> -- >> Post: opendatakit@googlegroups.com >> Unsubscribe: opendatakit+unsubscribe@googlegroups.com >> Options: http://groups.google.com/group/opendatakit?hl=en >> >

As Waylon said, not currently possible.

You might want to reconsider how to address your concern. Are you
collecting data with unique identifiers (asset tags, serial numbers, etc)?
Perhaps the sequential numbers are not necessary if each object is already
unique, capture/record the tag #, add your attribute data.

You'd have to discard your pre-conceived notions of how ODK works based on
your ArcPAD experience. Apples and oranges.

/r
Nik

··· On Thursday, October 25, 2012, Heath Smith wrote:

I'm use to collecting data on ArcPAD in which the database is localized on
the machine till we download it. We won't have cell service at our field
site so each phone will serve as a single database until we download it at
camp. My hope was that the form could search the phone for the last record
and add 1 to that records number, giving the new record the next number. It
would matter to us that other phones will have the same record on them
because when we download it we'll be able to see which phone that record
came from.

H

On Wednesday, October 24, 2012 11:35:31 PM UTC-7, ニコノコ wrote:

I think, in the ODK convention, a "record" refers to the whole dataset
captured in a form. 1 form entry = 1 record. Trying to number these records
consecutively will be an exercise in futility since a form wouldn't have an
idea how many other forms out there are collecting records.

Now, if you're collecting a list of items within the one form, you can
number these each consecutively but once you finalize and save the form,
your numbers reset back to the start.

So, which one is your use case?

/r
Nik

On Thursday, October 25, 2012, Heath Smith wrote:

Hi,

I just started building a form for data collection. I'm most familiar
with using forms in ArcPAD so this is a little new for me. I'm wanting to
generate a sequential number for each new record. Is there an simple way of
having the form add 1 to the previous record? I've done this in ArcPAD but
couldn't find much info here or on simple google searches.

Thanks,
Heath

--
Post: opendatakit@googlegroups.com
Unsubscribe: opendatakit+unsubscribe@**googlegroups.com
Options: http://groups.google.com/**group/opendatakit?hl=enhttp://groups.google.com/group/opendatakit?hl=en

--
Post: opendatakit@googlegroups.com <javascript:_e({}, 'cvml',
'opendatakit@googlegroups.com');>
Unsubscribe: opendatakit+unsubscribe@googlegroups.com <javascript:_e({},
'cvml', 'opendatakit%2Bunsubscribe@googlegroups.com');>
Options: http://groups.google.com/group/opendatakit?hl=en

Audrey,

Thanks for the question. There is not a comprehensive list of changes
as we are still actively deciding which features make the cut and
which don't.

The current rough estimate of a schedule is for ODK Collect 2.0
private alpha to be made available to developers who have open source
Open Data Kit complimentary tools before the end of the month. For
example we want to get feedback from the formhub developers on
protocols etc. After we gather feedback and make the necessary changes
we are hoping to have a public alpha available early January. This is
a rough estimate/guess and is subject to change.

When we prepare the public alpha we will have documentation to go with it.

However, we do not recommend people use the public alpha in field
deployments as things may change and there will probably still be some
bugs. The alphas are designed to allow people to give the ODK team
feedback.

Waylon

··· On Mon, Nov 12, 2012 at 8:35 PM, wrote: > Hi Waylon, > > Is there a comprehensive list somewhere outlining all of the proposed changes in version 2.0? Do you know when this will be released? Are there any alpha versions to play with yet? > > Thanks so much for your help! > > Best, > Audrey Henkels > > > On Thursday, October 25, 2012 9:20:20 AM UTC+3, Waylon Brunette wrote: >> Not without some custom code. Collect will automatically provide a >> >> unique identifier for a survey instance but not a sequential one. >> >> >> >> The problem is how to keep sequential numbers across multiple phones >> >> accurate as what does it mean if someone survey's person 1 on phone 1 >> >> and then survey with person 1 on phone 2? The numbers won't be >> >> sequential. It becomes application dependent on how you want to handle >> >> sequential numbers and what does it mean to have a sequential number >> >> does it refer to the phone? a person being interviewed? time of day? >> >> what happens when there are multiple phones and their sequence numbers >> >> collide? While there are solutions to this problem we have not found a >> >> good common solution that works for most use cases. People seem to >> >> diverge in how things should be handled so we have not added this >> >> functionality in yet to the common core and instead let people >> >> implement their definition of sequential to the code base. >> >> >> >> If you do not have the technical ability in your organization to make >> >> the necessary change may I suggest looking at the companies that are >> >> willing to be hired to help on our website. >> >> http://opendatakit.org/help/help-for-hire/ >> >> >> >> We plan to add something to make this easier in ODK 2.0 but that is >> >> still months off from being ready. >> >> >> >> There has been some previous discussion on the forum before on this: >> >> https://groups.google.com/forum/#!msg/opendatakit/uXaz3Ku97UU/8mDu606rJ7MJ >> >> >> >> On Wed, Oct 24, 2012 at 10:58 PM, Heath Smith wrote: >> >> > Hi, >> >> > >> >> > I just started building a form for data collection. I'm most familiar with >> >> > using forms in ArcPAD so this is a little new for me. I'm wanting to >> >> > generate a sequential number for each new record. Is there an simple way of >> >> > having the form add 1 to the previous record? I've done this in ArcPAD but >> >> > couldn't find much info here or on simple google searches. >> >> > >> >> > Thanks, >> >> > Heath >> >> > >> >> > -- >> >> > Post: opendatakit@googlegroups.com >> >> > Unsubscribe: opendatakit+unsubscribe@googlegroups.com >> >> > Options: http://groups.google.com/group/opendatakit?hl=en > > -- > Post: opendatakit@googlegroups.com > Unsubscribe: opendatakit+unsubscribe@googlegroups.com > Options: http://groups.google.com/group/opendatakit?hl=en

The sequence the submissions were received can already be determined by the
submission time.

With regards to adding a sequence number, yes you can do this but it will
require code changes....

This is actually more complicated than it sounds as a webserver can have
multiple instances active at any one time. Therefore, you need to save the
sequence number in the database and lock the sequence number so only one
one active session can update the sequence. There are other issues too but
those depend on the platform you are running ODK aggregate on. This will
also make Aggregate more inefficient and slower.

Waylon

··· On Tue, Jun 4, 2013 at 2:01 PM, Bashir Jahed wrote:

As a side note the this, if we wanted to add sequential numbers to
submissions based on sequence forms have been submitted to agregate, would
this be possible?

I.E. Each new record in ODK aggregate gets a sequential number on
submission?

On Thursday, 25 October 2012 07:58:13 UTC+2, Heath Smith wrote:

Hi,

I just started building a form for data collection. I'm most familiar
with using forms in ArcPAD so this is a little new for me. I'm wanting to
generate a sequential number for each new record. Is there an simple way of
having the form add 1 to the previous record? I've done this in ArcPAD but
couldn't find much info here or on simple google searches.

Thanks,
Heath

--

Post: opendatakit@googlegroups.com
Unsubscribe: opendatakit+unsubscribe@googlegroups.com
Options: http://groups.google.com/group/opendatakit?hl=en


You received this message because you are subscribed to the Google Groups
"ODK Community" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to opendatakit+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

I agree I need to discard my pre-conceived notions, just wanted to check if
it was possible. We will have a capture record but that was basically the
number generated by the record. :slight_smile:

Cheers!

··· On Thursday, October 25, 2012 12:05:23 AM UTC-7, ニコノコ wrote: > > As Waylon said, not currently possible. > > You might want to reconsider how to address your concern. Are you > collecting data with unique identifiers (asset tags, serial numbers, etc)? > Perhaps the sequential numbers are not necessary if each object is already > unique, capture/record the tag #, add your attribute data. > > You'd have to discard your pre-conceived notions of how ODK works based on > your ArcPAD experience. Apples and oranges. > > /r > Nik > > > On Thursday, October 25, 2012, Heath Smith wrote: > >> I'm use to collecting data on ArcPAD in which the database is localized >> on the machine till we download it. We won't have cell service at our field >> site so each phone will serve as a single database until we download it at >> camp. My hope was that the form could search the phone for the last record >> and add 1 to that records number, giving the new record the next number. It >> would matter to us that other phones will have the same record on them >> because when we download it we'll be able to see which phone that record >> came from. >> >> H >> >> On Wednesday, October 24, 2012 11:35:31 PM UTC-7, ニコノコ wrote: >>> >>> I think, in the ODK convention, a "record" refers to the whole dataset >>> captured in a form. 1 form entry = 1 record. Trying to number these records >>> consecutively will be an exercise in futility since a form wouldn't have an >>> idea how many other forms out there are collecting records. >>> >>> Now, if you're collecting a list of items within the one form, you can >>> number these each consecutively but once you finalize and save the form, >>> your numbers reset back to the start. >>> >>> So, which one is your use case? >>> >>> /r >>> Nik >>> >>> On Thursday, October 25, 2012, Heath Smith wrote: >>> >>>> Hi, >>>> >>>> I just started building a form for data collection. I'm most familiar >>>> with using forms in ArcPAD so this is a little new for me. I'm wanting to >>>> generate a sequential number for each new record. Is there an simple way of >>>> having the form add 1 to the previous record? I've done this in ArcPAD but >>>> couldn't find much info here or on simple google searches. >>>> >>>> Thanks, >>>> Heath >>>> >>>> -- >>>> Post: opendatakit@googlegroups.com >>>> Unsubscribe: opendatakit+unsubscribe@**googlegroups.com >>>> Options: http://groups.google.com/**group/opendatakit?hl=en >>>> >>> -- >> Post: opendatakit@googlegroups.com >> Unsubscribe: opendatakit+unsubscribe@googlegroups.com >> Options: http://groups.google.com/group/opendatakit?hl=en >> >

Yes. Managing the allocation of those pre-assigned numbers across a fleet
of mobile devices, and, potentially, invalidating them or reallocating
them, is well beyond the scope of ODK Collect and definitely something I
would put into a separate app. A good use case for the external string or
integer widget.

Mitch

··· On Sun, Oct 28, 2012 at 11:58 PM, Steve White wrote:

There are actually cases (inventory) where you need to used a pre-assigned
set of numbers. I'm working on a simple android app that's let's you input
numbers (typically as a range) and then request from this set of pre loaded
numbers. It's simplistic but useful in this case.

Cheers,
Steve

--
Post: opendatakit@googlegroups.com
Unsubscribe: opendatakit+unsubscribe@googlegroups.com
Options: http://groups.google.com/group/opendatakit?hl=en

--
Mitch Sundt
Software Engineer
University of Washington
mitchellsundt@gmail.com

Audrey,

I am not sure. It would be at the end of the month if it happens. My
guess is the public alpha release will be delayed but I don't know
it's dependent on so many things. In short we are behind and the team
is currently spread around the world so it will be a week or two
before we set the final schedule.

We will be looking for testers in January but I am not sure how wide
of distribution we will make the first alpha.

Stay tuned.

Waylon

··· On Tue, Jan 8, 2013 at 7:48 PM, wrote: > Hi Waylon, > > Hope you are doing well. I am just following up on my earlier question. Will there still be a public alpha released this month? Any updates on the timeline would be much appreciated. > > Thanks so much! > > Best, > Audrey > > > On Tuesday, November 13, 2012 10:38:38 PM UTC+3, Waylon Brunette wrote: >> Audrey, >> >> >> >> Thanks for the question. There is not a comprehensive list of changes >> >> as we are still actively deciding which features make the cut and >> >> which don't. >> >> >> >> The current rough estimate of a schedule is for ODK Collect 2.0 >> >> private alpha to be made available to developers who have open source >> >> Open Data Kit complimentary tools before the end of the month. For >> >> example we want to get feedback from the formhub developers on >> >> protocols etc. After we gather feedback and make the necessary changes >> >> we are hoping to have a public alpha available early January. This is >> >> a rough estimate/guess and is subject to change. >> >> >> >> When we prepare the public alpha we will have documentation to go with it. >> >> >> >> However, we do not recommend people use the public alpha in field >> >> deployments as things may change and there will probably still be some >> >> bugs. The alphas are designed to allow people to give the ODK team >> >> feedback. >> >> >> >> Waylon >> >> >> >> On Mon, Nov 12, 2012 at 8:35 PM, wrote: >> >> > Hi Waylon, >> >> > >> >> > Is there a comprehensive list somewhere outlining all of the proposed changes in version 2.0? Do you know when this will be released? Are there any alpha versions to play with yet? >> >> > >> >> > Thanks so much for your help! >> >> > >> >> > Best, >> >> > Audrey Henkels >> >> > >> >> > >> >> > On Thursday, October 25, 2012 9:20:20 AM UTC+3, Waylon Brunette wrote: >> >> >> Not without some custom code. Collect will automatically provide a >> >> >> >> >> >> unique identifier for a survey instance but not a sequential one. >> >> >> >> >> >> >> >> >> >> >> >> The problem is how to keep sequential numbers across multiple phones >> >> >> >> >> >> accurate as what does it mean if someone survey's person 1 on phone 1 >> >> >> >> >> >> and then survey with person 1 on phone 2? The numbers won't be >> >> >> >> >> >> sequential. It becomes application dependent on how you want to handle >> >> >> >> >> >> sequential numbers and what does it mean to have a sequential number >> >> >> >> >> >> does it refer to the phone? a person being interviewed? time of day? >> >> >> >> >> >> what happens when there are multiple phones and their sequence numbers >> >> >> >> >> >> collide? While there are solutions to this problem we have not found a >> >> >> >> >> >> good common solution that works for most use cases. People seem to >> >> >> >> >> >> diverge in how things should be handled so we have not added this >> >> >> >> >> >> functionality in yet to the common core and instead let people >> >> >> >> >> >> implement their definition of sequential to the code base. >> >> >> >> >> >> >> >> >> >> >> >> If you do not have the technical ability in your organization to make >> >> >> >> >> >> the necessary change may I suggest looking at the companies that are >> >> >> >> >> >> willing to be hired to help on our website. >> >> >> >> >> >> http://opendatakit.org/help/help-for-hire/ >> >> >> >> >> >> >> >> >> >> >> >> We plan to add something to make this easier in ODK 2.0 but that is >> >> >> >> >> >> still months off from being ready. >> >> >> >> >> >> >> >> >> >> >> >> There has been some previous discussion on the forum before on this: >> >> >> >> >> >> https://groups.google.com/forum/#!msg/opendatakit/uXaz3Ku97UU/8mDu606rJ7MJ >> >> >> >> >> >> >> >> >> >> >> >> On Wed, Oct 24, 2012 at 10:58 PM, Heath Smith wrote: >> >> >> >> >> >> > Hi, >> >> >> >> >> >> > >> >> >> >> >> >> > I just started building a form for data collection. I'm most familiar with >> >> >> >> >> >> > using forms in ArcPAD so this is a little new for me. I'm wanting to >> >> >> >> >> >> > generate a sequential number for each new record. Is there an simple way of >> >> >> >> >> >> > having the form add 1 to the previous record? I've done this in ArcPAD but >> >> >> >> >> >> > couldn't find much info here or on simple google searches. >> >> >> >> >> >> > >> >> >> >> >> >> > Thanks, >> >> >> >> >> >> > Heath >> >> >> >> >> >> > >> >> >> >> >> >> > -- >> >> >> >> >> >> > Post: opendatakit@googlegroups.com >> >> >> >> >> >> > Unsubscribe: opendatakit+unsubscribe@googlegroups.com >> >> >> >> >> >> > Options: http://groups.google.com/group/opendatakit?hl=en >> >> > >> >> > -- >> >> > Post: opendatakit@googlegroups.com >> >> > Unsubscribe: opendatakit+unsubscribe@googlegroups.com >> >> > Options: http://groups.google.com/group/opendatakit?hl=en > > -- > Post: opendatakit@googlegroups.com > Unsubscribe: opendatakit+unsubscribe@googlegroups.com > Options: http://groups.google.com/group/opendatakit?hl=en

Assigning or requiring the assignment of sequential numbers at the time of
data collection
is a Really Bad Idea.

At the time you assign the value, it requires global knowledge of the
state of the system
. As Waylon describes, this involves global locks on
shared databases, connectivity with the central server at the time the
value is selected, and this all impacts performance.

The solution is to have a partitioned value, where you construct a string
or number by concatenating two parts, e.g., a counter (1, 2, 3...) and a
namespace identifier, such as a deviceId, email, username, simserial or
other value that is unique across all your data collection devices (
http://opendatakit.org/help/form-design/examples/#Property_values ). This
enables the counter to be managed independently (on a per-device basis), as
the namespace identifier ensures that the combined value will never collide
with any other value in the system.

This, however, introduces gaps in your submission sequence number, e.g., if
you have deviceID 'a21b21aa' and deviceID 'b2131132' so you would have
separate submission streams with (counter-deviceID, you would have ids of,
e.g.,

1-a21b21aa, 2-a21b21aa, 3-a21b21aa, 4-a21b21aa...
1-b2131132, 2-b2131132, 3-b2131132, 4-b2131132...

As an example of creating a sequential ordering out of such string-valued
sequences with gaps, within ODK Aggregate, once the data has been uploaded,
a static sequential order (1,2,3,...) is determined by first sorting the
data by its meta-date-marked-as-complete (ascending in time) and then by
meta-instance-id (ascending in string value). Because Google's data
store is only 'eventually consistent', we maintain a 3-second 'settling
period' between the time a submission is inserted into the server and the
time it is added to this ordering. This gives Google 3 seconds to make the
data we've requested to be stored become 'eventually consistent'.

This ordering is used when we publish data, e.g., to Google Spreadsheet.
The data in row 2 of the spreadsheet is always exactly the same data, each
time you publish all your accumulated data.

Sequential ordering is easy to achieve AFTER data collection because the
data is all centralized -- you no longer have a highly distributed system.

Mitch

··· On Tue, Jun 4, 2013 at 4:09 AM, W. Brunette wrote:

The sequence the submissions were received can already be determined by
the submission time.

With regards to adding a sequence number, yes you can do this but it will
require code changes....

This is actually more complicated than it sounds as a webserver can have
multiple instances active at any one time. Therefore, you need to save the
sequence number in the database and lock the sequence number so only one
one active session can update the sequence. There are other issues too but
those depend on the platform you are running ODK aggregate on. This will
also make Aggregate more inefficient and slower.

Waylon

On Tue, Jun 4, 2013 at 2:01 PM, Bashir Jahed admin@osilab.net wrote:

As a side note the this, if we wanted to add sequential numbers to
submissions based on sequence forms have been submitted to agregate, would
this be possible?

I.E. Each new record in ODK aggregate gets a sequential number on
submission?

On Thursday, 25 October 2012 07:58:13 UTC+2, Heath Smith wrote:

Hi,

I just started building a form for data collection. I'm most familiar
with using forms in ArcPAD so this is a little new for me. I'm wanting to
generate a sequential number for each new record. Is there an simple way of
having the form add 1 to the previous record? I've done this in ArcPAD but
couldn't find much info here or on simple google searches.

Thanks,
Heath

--

Post: opendatakit@googlegroups.com
Unsubscribe: opendatakit+unsubscribe@googlegroups.com
Options: http://groups.google.com/group/opendatakit?hl=en


You received this message because you are subscribed to the Google Groups
"ODK Community" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to opendatakit+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

--

Post: opendatakit@googlegroups.com
Unsubscribe: opendatakit+unsubscribe@googlegroups.com
Options: http://groups.google.com/group/opendatakit?hl=en


You received this message because you are subscribed to the Google Groups
"ODK Community" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to opendatakit+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

--
Mitch Sundt
Software Engineer
University of Washington
mitchellsundt@gmail.com

Hi Mitch

I was thinking the exact same concept. I don't need exact sequential
numbering and gaps are fine. What I am trying to achieve is to have a way
to trace back to submitted results by referencing these results directly on
the odk DB via an external application. I basically need a unique
submission id for each submission that is near accurate in sequence by
device and the method you refer to above will easily achieve this. I don,t
need any sequential ordering for non submitted forms and am only interested
in assigning these id's to data in aggregate post submission.

I am trying to achieve a scenario that allows me to email the interviewee
the the questionnaire in a formatted pdf post submission and be able to
reference this submission and print the referable ID on the emailed results.

Bash

··· On Tue, Jun 4, 2013 at 8:26 PM, Mitch Sundt wrote:

Assigning or requiring the assignment of sequential numbers at the time
of data collection
is a Really Bad Idea.

At the time you assign the value, it requires global knowledge of the
state of the system
. As Waylon describes, this involves global locks on
shared databases, connectivity with the central server at the time the
value is selected, and this all impacts performance.

The solution is to have a partitioned value, where you construct a string
or number by concatenating two parts, e.g., a counter (1, 2, 3...) and a
namespace identifier, such as a deviceId, email, username, simserial or
other value that is unique across all your data collection devices (
http://opendatakit.org/help/form-design/examples/#Property_values ).
This enables the counter to be managed independently (on a per-device
basis), as the namespace identifier ensures that the combined value will
never collide with any other value in the system.

This, however, introduces gaps in your submission sequence number, e.g.,
if you have deviceID 'a21b21aa' and deviceID 'b2131132' so you would have
separate submission streams with (counter-deviceID, you would have ids of,
e.g.,

1-a21b21aa, 2-a21b21aa, 3-a21b21aa, 4-a21b21aa...
1-b2131132, 2-b2131132, 3-b2131132, 4-b2131132...

As an example of creating a sequential ordering out of such string-valued
sequences with gaps, within ODK Aggregate, once the data has been uploaded,
a static sequential order (1,2,3,...) is determined by first sorting the
data by its meta-date-marked-as-complete (ascending in time) and then by
meta-instance-id (ascending in string value). Because Google's data
store is only 'eventually consistent', we maintain a 3-second 'settling
period' between the time a submission is inserted into the server and the
time it is added to this ordering. This gives Google 3 seconds to make the
data we've requested to be stored become 'eventually consistent'.

This ordering is used when we publish data, e.g., to Google Spreadsheet.
The data in row 2 of the spreadsheet is always exactly the same data, each
time you publish all your accumulated data.

Sequential ordering is easy to achieve AFTER data collection because
the data is all centralized -- you no longer have a highly distributed
system.

Mitch

On Tue, Jun 4, 2013 at 4:09 AM, W. Brunette wbrunette@gmail.com wrote:

The sequence the submissions were received can already be determined by
the submission time.

With regards to adding a sequence number, yes you can do this but it will
require code changes....

This is actually more complicated than it sounds as a webserver can have
multiple instances active at any one time. Therefore, you need to save the
sequence number in the database and lock the sequence number so only one
one active session can update the sequence. There are other issues too but
those depend on the platform you are running ODK aggregate on. This will
also make Aggregate more inefficient and slower.

Waylon

On Tue, Jun 4, 2013 at 2:01 PM, Bashir Jahed admin@osilab.net wrote:

As a side note the this, if we wanted to add sequential numbers to
submissions based on sequence forms have been submitted to agregate, would
this be possible?

I.E. Each new record in ODK aggregate gets a sequential number on
submission?

On Thursday, 25 October 2012 07:58:13 UTC+2, Heath Smith wrote:

Hi,

I just started building a form for data collection. I'm most familiar
with using forms in ArcPAD so this is a little new for me. I'm wanting to
generate a sequential number for each new record. Is there an simple way of
having the form add 1 to the previous record? I've done this in ArcPAD but
couldn't find much info here or on simple google searches.

Thanks,
Heath

--

Post: opendatakit@googlegroups.com
Unsubscribe: opendatakit+unsubscribe@googlegroups.com
Options: http://groups.google.com/group/opendatakit?hl=en


You received this message because you are subscribed to the Google
Groups "ODK Community" group.
To unsubscribe from this group and stop receiving emails from it, send
an email to opendatakit+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

--

Post: opendatakit@googlegroups.com
Unsubscribe: opendatakit+unsubscribe@googlegroups.com
Options: http://groups.google.com/group/opendatakit?hl=en


You received this message because you are subscribed to the Google Groups
"ODK Community" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to opendatakit+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

--
Mitch Sundt
Software Engineer
University of Washington
mitchellsundt@gmail.com

--

Post: opendatakit@googlegroups.com
Unsubscribe: opendatakit+unsubscribe@googlegroups.com
Options: http://groups.google.com/group/opendatakit?hl=en


You received this message because you are subscribed to the Google Groups
"ODK Community" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to opendatakit+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

--
Bashir Jahed - Director
The OSi LAB (PTY) Ltd
www.osilab.net | 083 414 0453 | bashir@osilab.net

Where can we sign up for testing?

··· On Tue, Jan 8, 2013 at 4:04 PM, W. Brunette wrote:

Audrey,

I am not sure. It would be at the end of the month if it happens. My
guess is the public alpha release will be delayed but I don't know
it's dependent on so many things. In short we are behind and the team
is currently spread around the world so it will be a week or two
before we set the final schedule.

We will be looking for testers in January but I am not sure how wide
of distribution we will make the first alpha.

Stay tuned.

Waylon

On Tue, Jan 8, 2013 at 7:48 PM, audrey.henkels@gmail.com wrote:

Hi Waylon,

Hope you are doing well. I am just following up on my earlier question.
Will there still be a public alpha released this month? Any updates on the
timeline would be much appreciated.

Thanks so much!

Best,
Audrey

On Tuesday, November 13, 2012 10:38:38 PM UTC+3, Waylon Brunette wrote:

Audrey,

Thanks for the question. There is not a comprehensive list of changes

as we are still actively deciding which features make the cut and

which don't.

The current rough estimate of a schedule is for ODK Collect 2.0

private alpha to be made available to developers who have open source

Open Data Kit complimentary tools before the end of the month. For

example we want to get feedback from the formhub developers on

protocols etc. After we gather feedback and make the necessary changes

we are hoping to have a public alpha available early January. This is

a rough estimate/guess and is subject to change.

When we prepare the public alpha we will have documentation to go with
it.

However, we do not recommend people use the public alpha in field

deployments as things may change and there will probably still be some

bugs. The alphas are designed to allow people to give the ODK team

feedback.

Waylon

On Mon, Nov 12, 2012 at 8:35 PM, audrey.henkels@gmail.com wrote:

Hi Waylon,

Is there a comprehensive list somewhere outlining all of the proposed
changes in version 2.0? Do you know when this will be released? Are there
any alpha versions to play with yet?

Thanks so much for your help!

Best,

Audrey Henkels

On Thursday, October 25, 2012 9:20:20 AM UTC+3, Waylon Brunette wrote:

Not without some custom code. Collect will automatically provide a

unique identifier for a survey instance but not a sequential one.

The problem is how to keep sequential numbers across multiple phones

accurate as what does it mean if someone survey's person 1 on phone
1

and then survey with person 1 on phone 2? The numbers won't be

sequential. It becomes application dependent on how you want to
handle

sequential numbers and what does it mean to have a sequential number

does it refer to the phone? a person being interviewed? time of day?

what happens when there are multiple phones and their sequence
numbers

collide? While there are solutions to this problem we have not found
a

good common solution that works for most use cases. People seem to

diverge in how things should be handled so we have not added this

functionality in yet to the common core and instead let people

implement their definition of sequential to the code base.

If you do not have the technical ability in your organization to make

the necessary change may I suggest looking at the companies that are

willing to be hired to help on our website.

http://opendatakit.org/help/help-for-hire/

We plan to add something to make this easier in ODK 2.0 but that is

still months off from being ready.

There has been some previous discussion on the forum before on this:

https://groups.google.com/forum/#!msg/opendatakit/uXaz3Ku97UU/8mDu606rJ7MJ

On Wed, Oct 24, 2012 at 10:58 PM, Heath Smith heath623@gmail.com wrote:

Hi,

I just started building a form for data collection. I'm most
familiar with

using forms in ArcPAD so this is a little new for me. I'm wanting
to

generate a sequential number for each new record. Is there an
simple way of

having the form add 1 to the previous record? I've done this in
ArcPAD but

couldn't find much info here or on simple google searches.

Thanks,

Heath

--

Post: opendatakit@googlegroups.com

Unsubscribe: opendatakit+unsubscribe@googlegroups.com

Options: http://groups.google.com/group/opendatakit?hl=en

--

Post: opendatakit@googlegroups.com

Unsubscribe: opendatakit+unsubscribe@googlegroups.com

Options: http://groups.google.com/group/opendatakit?hl=en

--
Post: opendatakit@googlegroups.com
Unsubscribe: opendatakit+unsubscribe@googlegroups.com
Options: http://groups.google.com/group/opendatakit?hl=en

--
Post: opendatakit@googlegroups.com
Unsubscribe: opendatakit+unsubscribe@googlegroups.com
Options: http://groups.google.com/group/opendatakit?hl=en

--

http://www.arnalich.com/
Santiago Arnalich
Consultant, lecturer and author
+34 671700686
skype: sarnalich
...
www.arnalich.com
Consultancy & training for Development

The example form here http://opendatakit.org/help/form-design/ and the
XLSForm tool synthesize an instanceID field that is based upon a UUID.

I don't see documentation on how to provide your own calculate expression
for the instanceID in XLSForm, but you can change that to be anything you
want -- e.g., by adding starttime and deviceid to the metadata, you can do:



Mitch

··· On Tue, Jun 4, 2013 at 12:16 PM, Bashir Jahed wrote:

Hi Mitch

I was thinking the exact same concept. I don't need exact sequential
numbering and gaps are fine. What I am trying to achieve is to have a way
to trace back to submitted results by referencing these results directly on
the odk DB via an external application. I basically need a unique
submission id for each submission that is near accurate in sequence by
device and the method you refer to above will easily achieve this. I don,t
need any sequential ordering for non submitted forms and am only interested
in assigning these id's to data in aggregate post submission.

I am trying to achieve a scenario that allows me to email the interviewee
the the questionnaire in a formatted pdf post submission and be able to
reference this submission and print the referable ID on the emailed results.

Bash

On Tue, Jun 4, 2013 at 8:26 PM, Mitch Sundt mitchellsundt@gmail.comwrote:

Assigning or requiring the assignment of sequential numbers at the time
of data collection
is a Really Bad Idea.

At the time you assign the value, it requires global knowledge of the
state of the system
. As Waylon describes, this involves global locks on
shared databases, connectivity with the central server at the time the
value is selected, and this all impacts performance.

The solution is to have a partitioned value, where you construct a string
or number by concatenating two parts, e.g., a counter (1, 2, 3...) and a
namespace identifier, such as a deviceId, email, username, simserial or
other value that is unique across all your data collection devices (
http://opendatakit.org/help/form-design/examples/#Property_values ).
This enables the counter to be managed independently (on a per-device
basis), as the namespace identifier ensures that the combined value will
never collide with any other value in the system.

This, however, introduces gaps in your submission sequence number, e.g.,
if you have deviceID 'a21b21aa' and deviceID 'b2131132' so you would have
separate submission streams with (counter-deviceID, you would have ids of,
e.g.,

1-a21b21aa, 2-a21b21aa, 3-a21b21aa, 4-a21b21aa...
1-b2131132, 2-b2131132, 3-b2131132, 4-b2131132...

As an example of creating a sequential ordering out of such string-valued
sequences with gaps, within ODK Aggregate, once the data has been uploaded,
a static sequential order (1,2,3,...) is determined by first sorting the
data by its meta-date-marked-as-complete (ascending in time) and then by
meta-instance-id (ascending in string value). Because Google's data
store is only 'eventually consistent', we maintain a 3-second 'settling
period' between the time a submission is inserted into the server and the
time it is added to this ordering. This gives Google 3 seconds to make the
data we've requested to be stored become 'eventually consistent'.

This ordering is used when we publish data, e.g., to Google Spreadsheet.
The data in row 2 of the spreadsheet is always exactly the same data, each
time you publish all your accumulated data.

Sequential ordering is easy to achieve AFTER data collection because
the data is all centralized -- you no longer have a highly distributed
system.

Mitch

On Tue, Jun 4, 2013 at 4:09 AM, W. Brunette wbrunette@gmail.com wrote:

The sequence the submissions were received can already be determined by
the submission time.

With regards to adding a sequence number, yes you can do this but it
will require code changes....

This is actually more complicated than it sounds as a webserver can have
multiple instances active at any one time. Therefore, you need to save the
sequence number in the database and lock the sequence number so only one
one active session can update the sequence. There are other issues too but
those depend on the platform you are running ODK aggregate on. This will
also make Aggregate more inefficient and slower.

Waylon

On Tue, Jun 4, 2013 at 2:01 PM, Bashir Jahed admin@osilab.net wrote:

As a side note the this, if we wanted to add sequential numbers to
submissions based on sequence forms have been submitted to agregate, would
this be possible?

I.E. Each new record in ODK aggregate gets a sequential number on
submission?

On Thursday, 25 October 2012 07:58:13 UTC+2, Heath Smith wrote:

Hi,

I just started building a form for data collection. I'm most familiar
with using forms in ArcPAD so this is a little new for me. I'm wanting to
generate a sequential number for each new record. Is there an simple way of
having the form add 1 to the previous record? I've done this in ArcPAD but
couldn't find much info here or on simple google searches.

Thanks,
Heath

--

Post: opendatakit@googlegroups.com
Unsubscribe: opendatakit+unsubscribe@googlegroups.com
Options: http://groups.google.com/group/opendatakit?hl=en


You received this message because you are subscribed to the Google
Groups "ODK Community" group.
To unsubscribe from this group and stop receiving emails from it, send
an email to opendatakit+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

--

Post: opendatakit@googlegroups.com
Unsubscribe: opendatakit+unsubscribe@googlegroups.com
Options: http://groups.google.com/group/opendatakit?hl=en


You received this message because you are subscribed to the Google
Groups "ODK Community" group.
To unsubscribe from this group and stop receiving emails from it, send
an email to opendatakit+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

--
Mitch Sundt
Software Engineer
University of Washington
mitchellsundt@gmail.com

--

Post: opendatakit@googlegroups.com
Unsubscribe: opendatakit+unsubscribe@googlegroups.com
Options: http://groups.google.com/group/opendatakit?hl=en


You received this message because you are subscribed to the Google Groups
"ODK Community" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to opendatakit+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

--
Bashir Jahed - Director
The OSi LAB (PTY) Ltd
www.osilab.net | 083 414 0453 | bashir@osilab.net

--

Post: opendatakit@googlegroups.com
Unsubscribe: opendatakit+unsubscribe@googlegroups.com
Options: http://groups.google.com/group/opendatakit?hl=en


You received this message because you are subscribed to the Google Groups
"ODK Community" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to opendatakit+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

--
Mitch Sundt
Software Engineer
University of Washington
mitchellsundt@gmail.com